← All Cracked Math projects
PM-B04NOT STARTEDBEGINNERMATHPYTHONIMPACT ★☆☆☆☆

Finite differences vs the complex step: hunting the optimal h

Work out the best step size h before you measure it. Then watch rounding error draw a V on the error plot.

EST.5 h
REWARD+200 XP
AFTERSquire & Trapp 1998

ML-ARENA BENCHMARKS

PROOF: BENCH
METRICINDUSTRYMINE
Relative derivative error at the predicted-optimal h, and predicted vs measured optimal h
GOAL · Predicted optimal h within 10x of the measured one for every method and precision. The complex step reaches <= 1e-15 relative error on all analytic test functions (functions smooth enough to accept complex inputs).
Squire & Trapp 1998 complex-step method; IEEE fp64 eps = 2.2e-16 · measured locally—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

A finite difference estimates a derivative from function values at nearby points, for example (f(x+h) - f(x))/h. Too large an h gives truncation error (the error built into the approximation formula). Too small an h gives rounding error (floating-point noise that grows as h shrinks). Sweep h from 1e-1 down to 1e-20 for four methods: forward differences, central differences, the 5-point stencil (a formula that uses five nearby points) and the complex-step derivative (evaluate f at x + ih and read the derivative from the imaginary part, which avoids subtraction entirely). Use 10 test functions, including cancellation traps like (e^x - 1)/x, where subtracting two nearly equal numbers wipes out significant digits. Run everything in fp32 and fp64. For each method, predict the optimal h and the minimum error from machine epsilon (the relative gap between 1 and the next floating-point number) and the method's truncation order, before running anything. Then plot the error curves.

VICTORY CONDITIONS

  1. Predicted optimal h within 10x of the measured one for every method and precision. The complex step reaches <= 1e-15 relative error on all analytic test functions (functions smooth enough to accept complex inputs). (Relative derivative error at the predicted-optimal h, and predicted vs measured optimal h).
  2. A public repository with a README that shows your result and how to reproduce it.

Terms in plain words

Truncation error
The error a derivative formula makes even with exact arithmetic, because it uses a finite step h; it shrinks as h shrinks.
Machine epsilon
The smallest relative difference a floating-point format can represent, about 2.2e-16 for fp64; it sets the floor for rounding error.
Complex-step derivative
A trick that evaluates f at x plus a tiny imaginary step and takes the imaginary part divided by h, giving the derivative with no subtraction and so almost no rounding error.
Cancellation
The loss of correct digits when you subtract two nearly equal floating-point numbers.

Why it rates ★☆☆☆☆

Basic numerical hygiene. Hiring managers assume you have it rather than reward it.