Lecture 9 · Trustworthy Numerical Answers

When should we trust a numerical answer?¶

Last time: build a root finder and explain its stopping rule.

Today: propose → test → accept or revise → verify.

We will connect each advanced method to the problem it helps solve.

Recall · what does “small” mean?¶

• Residual magnitude: $|f(x_n)|$ measures equation mismatch; a signed residual keeps the sign.

• Update: $s_n=|x_n-x_{n-1}|$ measures the last move.

• Bracket: a valid $[a_n,b_n]$ containing $x_*$ bounds midpoint error:

Why go beyond the basic methods?¶

The basic methods show how approximation, iteration, and stopping work.

Advanced methods become useful when:

• A fast local step is unreliable from your starting guess.

• Each function evaluation is expensive.

• You fit many observations with a few parameters.

• An integrand has structure that uniform sampling resolves inefficiently.

Goal: recognize the problem, choose a method, and check its answer.

Ask what information the problem gives you¶

A local model of $f(x)$ proposes a step¶

The tangent line near $x_n$ is our local model of $f(x)$:

Separate direction from distance¶

• Direction: Newton supplies $p$.

• Step length: a line search chooses $\alpha$.

Trial rule: try the full step; shorten it if the acceptance test fails.

Define progress before accepting a step¶

Use a residual objective: $\phi(x)=\tfrac12 f(x)^2$.

For an exact Newton direction at a nonroot,

A short enough step decreases this objective under the differentiability assumptions. A full step may not.

A half-step can be more useful¶

Demonstration · compare the trial steps¶

For $f(x)=\arctan x$ at $x=2$, the initial residual magnitude is 1.107.

Backtracking accepts the first passing trial. It need not find the best step.

When is a line search useful?¶

• Use when: a local direction is promising but the full step overshoots.

• Example: nonlinear calibration from an uncertain starting guess.

• Check: actual objective decrease, not just the tangent’s prediction.

• Cost: rejected trials require extra evaluations.

• Limit: shorter steps cannot create a root. Check the domain, residual, and stopping conditions.

A bracket gives a different kind of protection¶

Keep opposite signs. A fallback bisection step reduces the bracket width.

Safeguarded methods · fast proposals, controlled progress¶

Use a hybrid method when you have a valid bracket and want faster progress than bisection alone.

1. Propose a fast step using local information.

2. Check that it is safe and makes adequate progress.

3. Otherwise, take a bracket-reducing step.

Brent-type methods combine interpolation and bisection; the preceding tangent example illustrates the safeguard idea, not Brent’s exact algorithm.

Continuity and reliable signs still matter.

From roots to fitting data¶

A residual is prediction minus observation: $r_i(\theta)=g(t_i;\theta)-y_i$.

Choose parameters $\theta$ to reduce $\Phi(\theta)=\tfrac12\sum_i r_i(\theta)^2$.

Noisy observations generally cannot all have zero residual.

The Jacobian · one slope for each input–output pair¶

Let $r(\theta)$ have $m$ outputs and $\theta$ have $n$ parameters.

• Row: one output’s sensitivity to all parameters.

• Column: all outputs’ sensitivity to one parameter.

A partial derivative changes one parameter while holding the others fixed.

The matrix is a local input–output map¶

For a small parameter update $p$,

The dimensions tell us what this means:

Component by component: $\Delta r_i\approx\sum_j J_{ij}p_j$.

Each entry has units “output $i$ per unit of parameter $j$.”

Example · three observations, two parameters¶

Fit $g(t;A,k)=Ae^{-kt}$ to fixed observations $y_i$ at $t=0,1,2$.

For $r_i=Ae^{-kt_i}-y_i$,

Evaluate the slopes at the current parameters¶

Use the Jacobian to predict a change¶

Increase $A$ by 0.10 and $k$ by 0.02:

These are predicted changes in residuals, not the residuals themselves.

Increasing amplitude raises the curve; increasing decay rate lowers later values. The effects partly cancel.

Check the local prediction against the actual change¶

For $p=(0.10,0.02)$, evaluate the exponential again:

For $0.1p$, the largest discrepancy falls from $7.91\times10^{-4}$ to $7.99\times10^{-6}$: about $99\times$ smaller.

Read the least-squares notation¶

DS/ML · from many outputs to one loss¶

For least-squares loss $\Phi(\theta)=\tfrac12 r(\theta)^{\mathsf T}r(\theta)$,

The Jacobian describes how parameters affect predictions; the loss gradient combines those effects into one direction of change.

Where it appears: nonlinear regression, calibration, and sensitivity of model predictions.

Backpropagation applies the chain rule through a model. It can compute products such as $J^{\mathsf T}r$ without storing the full Jacobian.

This connection does not mean neural networks are usually trained with Levenberg–Marquardt.

Computational science · coupled equations¶

A steady state or implicit time step can require solving $F(u)=0$ for many unknowns.

Linearize the coupled equations:

For a square, nonsingular Jacobian, Newton’s proposal solves

Examples: chemical equilibrium and coupled transport equations. Inverse problems instead often fit observations by minimizing a loss.

The Jacobian shows which variables affect which equations. Weak or dependent sensitivities can make the solve difficult.

Gauss–Newton · fit the local residual model¶

For a root, Newton sets a local approximation to zero.

For a fit, Gauss–Newton minimizes its squared mismatch:

At a minimizer, $J^{\mathsf T}Jp=-J^{\mathsf T}r$.

A local fit can propose a large step where the approximation is poor.

Derive a penalty on the proposed step¶

1. Define the penalized local fit in scaled parameters:

2. Differentiate with respect to the update $p$:

3. Solve where the gradient is zero:

The penalty limits the current step; it does not require parameters near zero.

Levenberg–Marquardt · damp the local fit¶

Local model: $r(\theta+p)\approx r(\theta)+Jp$.

Levenberg step:

Marquardt variants add scaling.

Accept only after checking: evaluate the actual fit improvement.

Read the contour plot before comparing steps¶

Contours: equal local squared mismatch.

$$q(p)=\tfrac12\lVert r+Jp\rVert^2$$

(21)

Cross: the local-model minimum.

Arrows: alternative damping choices from the same starting point, not successive iterations.

Damping changes the proposed direction¶

Small damping: near the Gauss–Newton step when $J^{\mathsf T}J$ is nonsingular.

Large damping: $p\approx-J^{\mathsf T}r/\lambda$, a small gradient-like step.

When is Levenberg–Marquardt useful?¶

• Use when: fitting smooth nonlinear models by least squares.

• Example: calibration, decay curves, or parameter fitting.

• Check: actual loss reduction, residual patterns, and parameter sensitivity.

• Limit: a small residual does not establish unique or well-determined parameters.

Trial rule: scale parameters sensibly; adjust damping after checking each trial.

Decision case · fitting a decay curve¶

You fit noisy observations with $Ae^{-kt}$ from a poor initial guess.

• Poor local prediction: damp the trial; compare predicted and actual loss reduction.

• Different parameter scales: choose meaningful scales before applying $\lambda I$.

• Only very early observations: sensitivity to $k$ is weak because $\partial g/\partial k=-At e^{-kt}$.

A stable numerical step does not establish that the data identify both parameters.

Three different ways to control a step¶

• Line search: shorten a direction until it produces sufficient decrease.

• Bracket safeguard: retain a valid bracket and replace unsafe proposals.

• Levenberg damping: change the linear system proposing the direction.

Invariant: only the bracketed method carries a root bracket.

Integration · check the error model too¶

Adaptive quadrature: refine where local estimates indicate difficulty.

Richardson: cancel a leading error term when its expansion applies.

Smaller reported error is evidence to investigate, not an unconditional certificate.

Where should we spend the next evaluation?¶

• Uniform refinement: sample everywhere.

• Local refinement: concentrate samples where needed.

The partition uses a known feature location; it is not an adaptive solver trace.

Agreement can mean that both estimates missed it¶

Use known feature locations, interval splitting, and independent checks. An error estimator only sees what was sampled.

Richardson · use the error pattern¶

1. Assume $Q(h)=I+Ch^p+\text{higher-order terms}$.

2. Refine to $Q(h/2)=I+Ch^p/2^p+\cdots$.

3. Cancel the leading error:

Symbols: $h$ is spacing; $p$ is leading error order; $I$ is the integral.

Richardson works when the assumed pattern applies¶

Useful when a smooth problem has a stable error expansion at these resolutions.

Sharp features, insufficient resolution, or rounding can break that pattern.

Gauss–Legendre · choose nodes and weights¶

Nodes: $x_i$ specify where to evaluate. Weights: $w_i$ set each contribution.

Two evaluations · compare the answers¶

For $f(x)=x^2$, the exact integral on $[-1,1]$ is $2/3$.

Endpoint trapezoid: $Q_T=f(-1)+f(1)=2$.

Two-node Gauss–Legendre:

With $n$ nodes, Gauss–Legendre is exact for polynomials through degree $2n-1$ in exact arithmetic.

Limit: two nodes handle cubics exactly, but not every function. What check would you use for a new integrand?

See what the two rules measure¶

• Left shading: trapezoid approximation.

• Right shading: true integral.

Gaussian weights combine sampled values; they are not automatically widths of little panels.

When is Gauss–Legendre useful?¶

• Use when: a smooth function can be evaluated at chosen nodes.

• Example: an expensive integrand on a finite interval.

• Check: compare orders or split the interval.

• Limit: no built-in error estimate; treat discontinuities and endpoint singularities explicitly.

Exactness: $n$ nodes integrate polynomials through degree $2n-1$ in exact arithmetic. This does not extend to every smooth function.

Fixed measured samples usually require a different approach.

Map the nodes to your interval¶

Library nodes $x_i$ and weights $w_i$ are defined on $[-1,1]$.

1. Map nodes to $[a,b]$:

2. Scale weights by half the interval length:

Convergence · which quantity contracts?¶

Estimate order from three errors¶

Suppose consecutive errors are $10^{-1},10^{-2},10^{-4}$.

Three errors support this estimate. A zero error cannot enter the logarithm.

A short triple suggests local behavior; it does not prove asymptotic order.

A bracket measures something different¶

Widths $1,1/2,1/4,1/8$ have ratio $1/2$.

The slope of log width against iteration is $-\log2$.

Actual midpoint errors need not decrease at every update.

Fast convergence is not the same as low cost¶

Compare work at the same meaningful accuracy.

Count:

• Function evaluations, including rejected trials.

• Derivative evaluations.

• Linear-system solves.

A higher-order quadrature rule uses more nodes per call. Cost also depends on what each evaluation computes.

Choose a method and a check¶

For one case, propose a method and an independent check:

1. A continuous scalar equation with a sign-changing bracket.

2. A nonlinear curve fitted to noisy observations.

3. An integral with a known narrow feature.

Project 3 will implement and apply bisection, Newton, and secant. Today’s advanced methods are an overview.

Project 2 · protected work time¶

Due Wednesday, September 23, 11:59 pm PT

Choose your first unresolved check:

• Separate cutoff error from quadrature refinement.

• Recover the uniform-disk limit.

• Explain convergence near a branch point.

• Compare adaptive accuracy with actual evaluation counts.

Show the predicted result, measured result, and the smallest case that disagrees.