The Nelson-Siegel model has four parameters: $\beta_0$, $\beta_1$, $\beta_2$, and $\lambda$. These parameters have the following restrictions:
- $\beta_0$, $\beta_1$ and $\beta_2$ can be any real numbers. They don't have specific restrictions, as they determine the level, slope, and curvature of the term structure. However, $\beta_0$ is usually positive since it represents the long-term yield level. Additionally, $\beta_1 + \beta_2 > 0$ might be required in order to guarantee that short term rates are positive.
- $\lambda$ should be positive, as you mentioned, to ensure the model's stability and smoothness.
The Nelson-Siegel-Svensson model extends the Nelson-Siegel model by introducing two additional parameters ($\beta_3$ and $\lambda_2$) to improve the model's flexibility. The restrictions for this model are:
- $\beta_0$, $\beta_1$, $\beta_2$ and $\beta_3$ can be any real numbers. Similar to the Nelson-Siegel model, $\beta_0$ is usually positive. $\beta_3$ determines the additional curvature of the term structure. As above, $\beta_0$ is usually positive since it represents the long-term yield level. Additionally, $\beta_1 + \beta_2 > 0$ might be required in order to guarantee that short term rates are positive.
- $\lambda_1$ and $\lambda_2$ should both be positive. They ensure the stability and smoothness of the model.
In practice, you can use optimization techniques (e.g., nonlinear least squares) to estimate the model parameters by minimizing the sum of squared errors (SE) between the actual yields and the approximated yields. The restrictions on $\lambda$ (in Nelson-Siegel) and $\lambda_1$ and $\lambda_2$ (in Nelson-Siegel-Svensson) can be enforced during the optimization process. In some cases, you may find it useful to apply additional constraints on the parameters based on the specific characteristics of the yield curve or market.
In the comments, you give an example of a particular term structure with an outlier on the first maturity. If I am not mistaken, your question is how to constrain Nelson-Siegel or Nelson-Siegel-Svensson in such a way that it is possible to calibrate it to problematic curves such as the given one.
However, constraining the model for such a custom situation will likely not succeed due to the difficulty of determining an initial starting point that does not lead to a local optimum and collinearity considerations. Rather the way forward lies in a different methodology.
For further reading on this, I would highly recommend Numerical Methods and Optimization in Finance, Chapter 14, specifically section 14.1.2 (1). An implementation is described in (2).
This implementation gives the following fit with the constraints mentioned above:
(1) Manfred Gilli, Dietmar Maringer, and Enrico Schumann. (2019). Numerical Methods and Optimization in Finance (2nd ed.). Academic Press.