Least squares and curve fitting

Fitting a curve by shrinking the errors

Many models are curves with a few parameters, and fitting means choosing the parameters that make the curve sit close to the data. scipy.optimize.least_squares(fun, x0) does this. You write fun(p), which returns the vector of residuals, one for each data point, where a residual is the model value minus the data value. The function then minimises the cost $\tfrac12\sum r_i^2$. It starts at your guess x0 and takes steps using the Jacobian $J$, the table of how each residual changes when each parameter changes. A Gauss–Newton step solves $J^{\top}J\,\delta=-J^{\top}r$, and then moves the parameters by $\delta$. For a model that is linear in its parameters one such step lands on the answer. bounds=(low, high) keeps parameters in a sensible range. curve_fit is a friendlier wrapper that takes the model $f(x,\text{params})$ directly.

Squared errors let one wild point dominate: a residual of $10$ counts $100$. A robust loss grows more slowly for large residuals. With loss='soft_l1' and f_scale=1 the penalty is $\rho(z)=2(\sqrt{1+z}-1)$, where $z$ is the squared residual. Fitting yield curves is the classic use. The Nelson–Siegel curve is $y(\tau)=\beta_0+\beta_1\frac{1-e^{-\tau/\lambda}}{\tau/\lambda}+\beta_2\Big(\frac{1-e^{-\tau/\lambda}}{\tau/\lambda}-e^{-\tau/\lambda}\Big)$. It has a long-run level $\beta_0$, a slope $\beta_1$, a hump $\beta_2$ and a decay length $\lambda$ in years, and four numbers describe the whole curve. After a fit, the noise level is estimated by $s=\sqrt{\mathrm{SSE}/(n-p)}$, with $n$ points and $p$ parameters.

• Residuals in, parameters out: fun(p) returns model minus data, and least_squares shrinks the sum of squares.
• Robust loss for outliers: soft_l1 stops one wild point from dragging the whole curve.
• Four numbers for a curve: Nelson–Siegel summarises a yield curve with a level, a slope, a hump and a decay length.

Lab · what's graded

• A slope through the origin10 pts
• A Gauss–Newton step15 pts
• A robust penalty15 pts
• The noise level after a fit10 pts
• The arguments of least_squares10 pts
• A curve-fitting workflow10 pts
• Why a robust loss30 pts