Brief
A finite difference estimates a derivative from function values at nearby points, for example (f(x+h) - f(x))/h. Too large an h gives truncation error (the error built into the approximation formula). Too small an h gives rounding error (floating-point noise that grows as h shrinks). Sweep h from 1e-1 down to 1e-20 for four methods: forward differences, central differences, the 5-point stencil (a formula that uses five nearby points) and the complex-step derivative (evaluate f at x + ih and read the derivative from the imaginary part, which avoids subtraction entirely). Use 10 test functions, including cancellation traps like (e^x - 1)/x, where subtracting two nearly equal numbers wipes out significant digits. Run everything in fp32 and fp64. For each method, predict the optimal h and the minimum error from machine epsilon (the relative gap between 1 and the next floating-point number) and the method's truncation order, before running anything. Then plot the error curves.
VICTORY CONDITIONS
- Predicted optimal h within 10x of the measured one for every method and precision. The complex step reaches <= 1e-15 relative error on all analytic test functions (functions smooth enough to accept complex inputs). (Relative derivative error at the predicted-optimal h, and predicted vs measured optimal h).
- A public repository with a README that shows your result and how to reproduce it.
Terms in plain words
- Truncation error
- The error a derivative formula makes even with exact arithmetic, because it uses a finite step h; it shrinks as h shrinks.
- Machine epsilon
- The smallest relative difference a floating-point format can represent, about 2.2e-16 for fp64; it sets the floor for rounding error.
- Complex-step derivative
- A trick that evaluates f at x plus a tiny imaginary step and takes the imaginary part divided by h, giving the derivative with no subtraction and so almost no rounding error.
- Cancellation
- The loss of correct digits when you subtract two nearly equal floating-point numbers.
Why it rates ★☆☆☆☆
Basic numerical hygiene. Hiring managers assume you have it rather than reward it.