SC-05

Course 5

Numerical Differentiation

Approximate derivatives from discrete data using finite difference formulas. Understand the trade-off between truncation and rounding errors, and learn higher-order methods for improved accuracy.

Learning Objectives

Derive finite difference formulas using Taylor series
Understand forward, backward, and central differences
Analyze truncation error and accuracy order
Balance truncation and rounding errors for optimal step size
Apply higher-order formulas for improved accuracy
Use interpolation polynomials for derivative computation

1. Basic Finite Difference Formulas

The derivative is defined as a limit: $f'(x) = \lim_{h \to 0} \frac{f(x+h) - f(x)}{h}$ . For numerical computation, we use finite $h$ .

Definition 5.1: Forward Difference

f'(x) \approx \frac{f(x+h) - f(x)}{h}

Using Taylor expansion: $f(x+h) = f(x) + hf'(x) + \frac{h^2}{2}f''(\xi)$

Error: $\frac{h}{2}f''(\xi) = O(h)$

Definition 5.2: Backward Difference

f'(x) \approx \frac{f(x) - f(x-h)}{h}

Error: $O(h)$

Definition 5.3: Central Difference

f'(x) \approx \frac{f(x+h) - f(x-h)}{2h}

Error: $-\frac{h^2}{6}f'''(\xi) = O(h^2)$

Theorem 5.1: Central Difference Derivation

From Taylor series:

f(x+h) = f(x) + hf'(x) + \frac{h^2}{2}f''(x) + \frac{h^3}{6}f'''(x) + O(h^4)

f(x-h) = f(x) - hf'(x) + \frac{h^2}{2}f''(x) - \frac{h^3}{6}f'''(x) + O(h^4)

Subtracting and dividing by $2h$ gives the central difference with $O(h^2)$ error.

Example: Comparing Difference Formulas

For $f(x) = e^x$ at $x = 0$ with $h = 0.1$ :

Method	Formula	Approximation	Error
Forward	$(e^{0.1} - 1)/0.1$	1.0517	0.0517
Backward	$(1 - e^{-0.1})/0.1$	0.9516	0.0484
Central	$(e^{0.1} - e^{-0.1})/0.2$	1.0017	0.0017

Exact value: $f'(0) = 1$ . Central difference is most accurate.

2. Second Derivative Formula

Definition 5.4: Central Difference for Second Derivative

f''(x) \approx \frac{f(x+h) - 2f(x) + f(x-h)}{h^2}

Error: $-\frac{h^2}{12}f^{(4)}(\xi) = O(h^2)$

Theorem 5.2: Derivation

Adding the Taylor expansions for $f(x+h)$ and $f(x-h)$ :

f(x+h) + f(x-h) = 2f(x) + h^2 f''(x) + \frac{h^4}{12}f^{(4)}(\xi) + O(h^6)

Solving for $f''(x)$ gives the formula.

Remark:

Higher derivatives can be approximated similarly, but numerical differentiation becomes increasingly unstable. Whenever possible, use analytical derivatives or automatic differentiation instead.

3. Error Analysis and Optimal Step Size

Numerical differentiation involves two error sources: truncation error (from the approximation) and rounding error (from finite-precision arithmetic).

Theorem 5.3: Total Error

For forward difference with rounding error $\epsilon$ in $f$ :

\text{Total Error} \approx \underbrace{\frac{h}{2}|f''(\xi)|}_{\text{truncation}} + \underbrace{\frac{2\epsilon}{h}}_{\text{rounding}}

Theorem 5.4: Optimal Step Size

Minimizing total error by differentiating with respect to $h$ :

h_{\text{opt}} = 2\sqrt{\frac{\epsilon}{|f''|}} \approx \sqrt{\epsilon}

For double precision ( $\epsilon \approx 10^{-16}$ ), optimal $h \approx 10^{-8}$ .

Example: Optimal Step Size

For central difference (error $O(h^2)$ ):

\text{Total Error} \approx \frac{h^2}{6}|f'''| + \frac{2\epsilon}{h}

h_{\text{opt}} \approx \left(\frac{3\epsilon}{|f'''|}\right)^{1/3} \approx \epsilon^{1/3} \approx 10^{-5}

Note:

Key insight: Making $h$ too small actually increases error! This is why numerical differentiation is called ill-conditioned.

4. Higher-Order Formulas

Definition 5.5: Five-Point Central Difference

f'(x) \approx \frac{-f(x+2h) + 8f(x+h) - 8f(x-h) + f(x-2h)}{12h}

Error: $O(h^4)$

Definition 5.6: Five-Point Second Derivative

f''(x) \approx \frac{-f(x+2h) + 16f(x+h) - 30f(x) + 16f(x-h) - f(x-2h)}{12h^2}

Error: $O(h^4)$

Theorem 5.5: Richardson Extrapolation

If $D(h)$ approximates $f'(x)$ with error $O(h^2)$ , then:

D^*(h) = \frac{4D(h/2) - D(h)}{3}

approximates $f'(x)$ with error $O(h^4)$ .

Example: Richardson Extrapolation Applied

For central difference $D(h) = \frac{f(x+h) - f(x-h)}{2h}$ :

$D(0.2) = 1.0067$

$D(0.1) = 1.0017$

$D^* = \frac{4(1.0017) - 1.0067}{3} = 1.00003$

The extrapolated value is much closer to the exact value of 1.

5. Interpolation-Based Differentiation

Given data at unequally-spaced points, construct an interpolating polynomial and differentiate it.

Definition 5.7: Differentiation via Interpolation

Given data $(x_0, f_0), \ldots, (x_n, f_n)$ , let $P_n(x)$ be the interpolating polynomial. Then:

f'(x) \approx P_n'(x)

Example: Three-Point Formula

For equally-spaced points $x_0, x_1 = x_0+h, x_2 = x_0+2h$ :

f'(x_0) \approx \frac{-3f_0 + 4f_1 - f_2}{2h}

f'(x_1) \approx \frac{-f_0 + f_2}{2h}

f'(x_2) \approx \frac{f_0 - 4f_1 + 3f_2}{2h}

All have error $O(h^2)$ .

Remark:

Endpoint formulas: At boundaries where central differences aren't available, use one-sided formulas. The three-point forward difference $\frac{-3f_0 + 4f_1 - f_2}{2h}$ achieves $O(h^2)$ accuracy at the left endpoint.

Practice Quiz

Numerical Differentiation Quiz

Questions

Correct

Accuracy

The forward difference approximation

f'(x) \approx \frac{f(x+h) - f(x)}{h}

has error:

Easy

Not attempted

The central difference

f'(x) \approx \frac{f(x+h) - f(x-h)}{2h}

has error:

Easy

Not attempted

For the second derivative, the formula

f''(x) \approx \frac{f(x+h) - 2f(x) + f(x-h)}{h^2}

has error:

Easy

Not attempted

Why is numerical differentiation considered ill-conditioned?

Medium

Not attempted

For optimal accuracy in numerical differentiation, the step size

h

should:

Medium

Not attempted

The five-point formula for

f'(x)

using

f(x-2h), f(x-h), f(x), f(x+h), f(x+2h)

has error:

Medium

Not attempted

Richardson extrapolation can improve a derivative approximation from

O(h^2)

to:

Hard

Not attempted

The backward difference

f'(x) \approx \frac{f(x) - f(x-h)}{h}

is preferred when:

Medium

Not attempted

Using polynomial interpolation through

(x_0, f_0), ..., (x_n, f_n)

, the derivative

f'(x_k)

can be computed by:

Hard

Not attempted

For equally-spaced data, the error in computing

f'(x_0)

using Lagrange interpolation on

n+1

points is:

Hard

Not attempted

Frequently Asked Questions

Why is numerical differentiation considered unstable?

Division by small $h$ amplifies any error in function values. If $f(x+h)$ and $f(x)$ each have rounding error $\epsilon$ , the error in $(f(x+h) - f(x))/h$ is roughly $2\epsilon/h$ .

As $h \to 0$ , this rounding error grows without bound, while truncation error decreases. There's an optimal $h$ that minimizes total error.

When should I use forward vs. central differences?

Central difference: Default choice for interior points (higher accuracy).

Forward/backward difference: Use at boundaries or in real-time applications where future data isn't available.

How do I choose the step size h?

For double precision: use $h \approx 10^{-8}$ for $O(h)$ methods, $h \approx 10^{-5}$ for $O(h^2)$ methods. These are rough guidelines; the optimal $h$ depends on the function's derivatives.

How does automatic differentiation compare?

Automatic differentiation (AD) computes exact derivatives (to machine precision) by applying the chain rule systematically. It avoids both truncation error (unlike finite differences) and symbolic complexity (unlike symbolic differentiation). AD is preferred when available.

Can I use complex step differentiation?

Yes! The complex step method uses $f'(x) \approx \text{Im}(f(x + ih))/h$ which has $O(h^2)$ accuracy without subtraction cancellation. You can use very small $h$ (like $10^{-100}$ ) without rounding error issues.