Lecture 8 · Finite Differences and Root-Finding Tools

From geometric steps to reusable code¶

1. Approximate a slope using function values and Taylor expansion.

2. Rebuild Newton’s update, then reuse information in secant.

3. Interpret small quantities before declaring convergence.

4. Build a module whose tests check the mathematical contract.

Jump to: Finite differences · Newton to secant · Stopping · Module activity · Advanced reference

Today’s question¶

How do we turn a root-finding formula into a result we can explain?

Recall quantile inversion: solve $C(x)-q=0$. Each new target probability $q$ requires another solve.

Equilibrium models and implicit ML predictions also reuse root finders. We need accuracy evidence and clear failure behavior.

Today: approximate a slope → reuse values → decide when to stop → build a tested module.

Recall bisection: keep a root bracketed¶

For continuous $f$, retain opposite endpoint signs to keep a root $r$ bracketed. Evaluate $m$; retain a half; repeat.

Can local shape suggest the next input?¶

Bisection uses signs to retain a bracket. Newton uses a local slope to propose a new input.

To use a slope, we first need to understand how a computer approximates a derivative from function values.

Route: approximate a derivative → rebuild Newton’s update → reuse samples in secant.

I · Approximate a derivative from values¶

What if we can evaluate $f$, but have no derivative routine?

Use two nearby values:

$$D_hf=\frac{\Delta f}{h}=\frac{f(x+h)-f(x)}{h}.$$

(2)

As $h\to0$, this secant slope tends to $f'(x)$ when the derivative exists.

A computer uses $h\ne0$. Taylor expansion explains the error.

Taylor explains the derivative error¶

For sufficiently smooth $f$ near $x$,

Subtract $f(x)$ and divide every remaining term by $h\ne0$:

When $f''(x)\ne0$, the leading error is first order: halving $h$ approximately halves it. Derivation reference.

Predict: is the smallest step best?¶

For $f(x)=e^x$ at zero, the exact derivative is 1.

# lecture-example: difference
import numpy as np
steps = 10.0 ** (-np.arange(1, 17))
slopes = (np.exp(steps) - 1.0) / steps
errors = np.abs(slopes - 1.0)
for h, slope, error in zip(steps[::3], slopes[::3], errors[::3]):
    print(f"h={h:.0e}: slope={slope:.8f}, error={error:.2e}")

arange excludes 17. np.abs measures error; zip pairs entries. [::3] prints every third step.

Predict: which step gives the smallest error? Run, then compare with the next plot.

Smaller helps until rounding interferes¶

Two errors pull the step size in opposite directions¶

II · From Newton to secant¶

Near the current estimate $x_k$, replace the curve by its tangent model:

Set $\ell_k(x_{k+1})=0$ and solve for the correction:

Here $k$ counts updates, $f(x_k)$ is the residual, and $f'(x_k)$ is the local slope. We solve the linear model exactly; the curve is only approximated.

Predict Newton’s next input¶

For $f(x)=x^2-2$, use $f'(x)=2x$ and start at $x_0=2$.

Predict: where does the tangent cross zero? Is the original curve also zero at that input?

What must we recompute before taking another step?

Watch Newton calculate, move, and rebuild¶

Open or download this short animation

Pause after the first update: why must we reevaluate the curve? Replay, then compare corrections. Explanation.

The next step needs a new tangent¶

At $x_1=3/2$, evaluate the curve again:

$$f(3/2)=1/4,\qquad f'(3/2)=3.$$

(10)

$$x_2=\frac32-\frac{1/4}{3}=\frac{17}{12}.$$

(11)

A nonzero derivative permits the update; it does not guarantee convergence. The new point must remain in the domain. Newton does not preserve a bracket.

Put our derivative approximation into Newton¶

When a derivative routine is unavailable, Part I supplies

Replace the derivative in Newton’s update:

We choose $h\ne0$ and evaluate an extra nearby sample $f(x_k+h)$. The slope estimate must be usable and nonzero; truncation and rounding affect it.

Next idea: we already know the previous iterate’s function value. Can we reuse it instead?

Secant obtains its slope from the iteration¶

Finite-difference Newton: choose a new sample near the current iterate.

Secant: reuse the previous iterate and its function value. The separation changes with the iteration; there is no separately prescribed $h$.

Secant reuses two iterates¶

Approximate Newton’s derivative using two previous points:

Substitute into Newton’s step:

A secant line joins two samples. The secant method specifically reuses successive iterates; their separation replaces a separately chosen $h$.

Predict the secant intersection¶

For $f(x)=x^2-2$, start with $x_0=1$ and $x_1=2$.

Use the update to find $x_2$. Which two points will define the following line?

Watch which two values secant keeps¶

Open or download this short animation

Pause after the first update: which point is retained, and which is discarded? Replay to check. Explanation.

Translate the geometry into an update¶

For the two points $(1,-1)$ and $(2,2)$, the slope is

The line through the newest point is $\ell(x)=2+3(x-2)$. Set $\ell(x)=0$: $x_2=4/3$.

Now evaluate $f(4/3)$ and retain the previous value $f(2)$ for the next line. Only one new function value is needed. Unlike bisection, we keep recent points rather than opposite signs.

Compare what must be evaluated¶

Count actual evaluations: derivatives may be cheap or expensive.

Distinct inputs can have equal function values¶

For $f(x)=x^2-2$, choose $x_0=-1$ and $x_1=1$:

The secant is horizontal at $y=-1$: it never crosses zero. The update divides by zero.

A tiny value difference can also produce an unreliable step. Report the failure; an arbitrary perturbation is not a guaranteed repair.

III · Decide what convergence means¶

Question: small in which quantity, and relative to which scale?

In Lecture 5, atol and rtol expressed a comparison allowance. Here we must also decide what evidence to compare with that allowance.

A root-error bound, a small update, and a small residual answer different questions. A solver’s stopping rule belongs in its documented interface.

Three different meanings of small¶

Bound: limits root error for a valid bracket. Step: measures movement. Residual: measures how well the equation is satisfied.

Root error and step have $x$’s units; residual has $f$’s units.

A small residual need not locate the root¶

A simple root has $f(r)=0$ and $f'(r)\ne0$. Near it, Taylor gives

For a stated function-error scale, dividing by a small slope amplifies position uncertainty. This is a local approximation, not a global error bound.

Predict: for $f(x)=10^{-12}(x-1)$ at $x=2$, what are the residual and root error?

Scaling the function changes the residual¶

Same root, different residual. Choose residual tolerances using the equation’s scale and units.

Optional live rescaling check.

Give tolerances a meaning¶

A root-location criterion can use

• atol: an absolute allowance in the units of $x$.

• rtol: a dimensionless relative allowance.

• Near zero, a nonzero absolute scale may be necessary.

Bisection compares a bound with $\tau_x$. Newton and secant compare their step, then inspect the residual separately. A tiny step can also mean stagnation.

In this lab, use the current evaluated midpoint $m$: compare its bracket bound with $\tau_m$.

Work out the allowance before running the solver¶

Suppose $x$ is measured in seconds. Choose, for illustration, $\mathrm{atol}=10^{-6}\,\mathrm{s}$ and $\mathrm{rtol}=10^{-6}$.

The same relative allowance permits a larger absolute difference at larger scales. Choose these values from the task’s accuracy needs; the example is not a universal default.

Predict: does the smaller bracket certify the old midpoint?¶

For $f(x)=x-0.9$ on $[0,1]$, take $\mathrm{atol}=0.25$ and $\mathrm{rtol}=0$.

The evaluated midpoint is $m=0.5$. Its negative value retains the bracket $[0.5,1]$, whose half-width is 0.25.

Predict: can we now return the already evaluated $m=0.5$ with a root-error bound of 0.25?

Which point is the center of the new bracket?

A bound belongs to a particular midpoint¶

Here the root is known. A general solver uses the current bracket bound to certify its midpoint.

IV · Build and test a root-finding module¶

Work from a mathematical invariant to an interface, then test both success and failure.

Core: implement and test bisection on a non-project equation.

Extension: add Newton or secant and compare iteration histories.

This module can carry into Project 3 when it is posted next week. Today does not introduce a separate graded submission.

Give the code an importable home¶

root-lab/
├── roots.py
├── tests/
│   └── test_roots.py
└── exploration.ipynb

Trace → implement → harden¶

Keep the required contract open while working. A first successful root is a checkpoint; complete all three stages before Newton or secant.

Trace first: explain each decision¶

Use $f(x)=x^2-2$ on $[1,2]$. Before coding, make three rows:

For each row, explain why the retained endpoints still have opposite signs. The bound belongs to the midpoint evaluated in that row.

Check: a partner can reconstruct your next bracket from your recorded evidence.

Implement: agree on the interface before the loop¶

def bisect(f, a, b, atol=1e-12, rtol=0.0, max_iter=200):
    """Return (root, n_iter, history); implement the method yourself."""
    raise NotImplementedError

History: evaluated midpoints, in order.

Count: n_iter == len(history).

Return: the last evaluated midpoint, history[-1].

Endpoint zero: return that endpoint with n_iter=0 and history=[].

Required lab contract: inputs and failures¶

• Endpoints: finite real scalars with $a<b$; handle endpoint zeros before the sign test.

• Tolerances: finite, nonnegative; at least one positive.

• max_iter: a positive integer, excluding Boolean values.

• Function values: finite real scalars; compare signs directly rather than multiplying values.

Use ValueError for invalid inputs, an invalid bracket, or unusable function values. Use RuntimeError for an exhausted iteration budget or an unmet tolerance when no interior midpoint is representable.

Continuity and reliable function signs are mathematical preconditions; input checks cannot establish them.

Required lab contract: stopping and recorded evidence¶

At the evaluated midpoint $m$ of the current bracket $[a,b]$, use

For an exact midpoint, $B_m=(b-a)/2$. The maximum distance also covers a rounded, off-center midpoint.

• Accept a zero computed residual or $B_m\le\tau_m$; record this midpoint before returning.

• Otherwise retain the sign-changing half and continue. Its new width does not certify the old midpoint.

• If evaluation underflows to zero or signs are unreliable, the numerical bracket is not a real-arithmetic guarantee.

Record evaluated midpoints only. Endpoint roots have an empty history.

Translate the reading interface deliberately¶

Both use a bracket-based position criterion. The lab also records evidence for inspection.

Shared exceptions: ValueError for invalid inputs; RuntimeError for non-convergence.

Reading and lab: evaluation conventions¶

Reading: may accept the width before evaluating the midpoint. Allows $a=b$ for an endpoint root.

Lab: requires $a<b$. Evaluate and record the accepted midpoint so root == history[-1].

Endpoint exception: return the endpoint with zero midpoint evaluations and an empty history.

Use the lab contract when writing your tests.

Implement the invariant, not just the update¶

1. Check inputs and endpoint roots; establish a valid bracket.

2. Evaluate its midpoint and record it.

3. Check a zero computed residual or the current midpoint error bound.

4. Otherwise retain the half with opposite endpoint signs.

5. Stop with a useful failure message if the iteration budget is exhausted.

Work on continuous functions. Input checks cannot prove continuity. Do not silently replace a missing bracket with a guessed interval.

Test a known root without solving the project¶

import numpy as np
from roots import bisect

def test_known_root():
    root, n_iter, history = bisect(
        lambda x: x*x - 2, 1.0, 2.0, atol=1e-8, rtol=0.0
    )
    assert abs(root - np.sqrt(2)) <= 1e-8
    assert n_iter == len(history)
    assert history[-1] == root

A lambda is a small anonymous function. Here it supplies the residual. Run from the lab directory with python -m pytest.

An expected failure is a passing test¶

import pytest
from roots import bisect

def test_invalid_bracket():
    with pytest.raises(ValueError):
        bisect(lambda x: x*x + 1, -1.0, 1.0)

pytest.raises passes only if the enclosed call raises the expected exception. Returning a number here is a failure of the solver’s contract.

Save: tests/test_roots.py. Run from root-lab/: python -m pytest.

After module edits, restart the notebook kernel and rerun imports.

Harden: test the contract beyond one root¶

Use the input and failure contract as your checklist:

• Reject invalid endpoints, tolerances, iteration limits, and unusable function values.

• Distinguish a missing bracket from exhausted iterations or numerical stagnation.

• Check endpoint history and the midpoint to which the bound belongs.

Check: each rejection has an expected exception test. Exercise the relative tolerance before adding another method.

Add tests that could expose a mistake¶

Trace three brackets on paper; compare signs and widths with your code. Checking only the final root misses invariant failures.

Test the relative term¶

Design a case where a nonzero rtol changes midpoint acceptance.

• Hold the function, bracket, and atol fixed; compare with rtol=0.

• Predict which run should stop first, then compare the histories.

• Explain why tests using only rtol=0 cannot detect an ignored relative term.

Extend only after bisection is hardened¶

After the required contract checks pass, add Newton or secant for the same equation.

• Plot $|x_k-\sqrt2|$ against iteration number.

• Count function and derivative evaluations separately.

• Try a second starting point; explain changed behavior.

A faster example is not a universal ranking. Next lecture develops convergence conditions, failure modes, and safeguards.

Build, test, and explain your module¶

Need a refresher? Use bisection’s mathematical foundation, module boundaries, and documentation and tests. Build from the invariant before comparing with the reading’s implementation; use today’s interface. Advanced root-finding sections are Lecture 9/reference.

Show your paper trace, passing behavior tests, and required failure checks. Explain what the stopping evidence establishes. Finish hardening bisection before starting an extension.

Keep your module and iteration evidence for Project 3. We continue root finding in Lecture 9; the scientific project is posted next week.

V · Reference for Lecture 9 and the lab¶

Keep the following material accessible for review and Lecture 9. The required lab contract is in the activity section above.