Brief
Your NumPy autograd from BL-11 runs one operation at a time. Each element-wise op reads its inputs from memory, writes a whole new array, and the next op reads that array back. For big arrays the time goes to moving data, not to maths. In BL-14 you saw the same thing from the other side: speed comes from how data moves. This project builds a tiny compiler that removes those round trips, the same idea behind torch.compile and XLA.
A compiler here has three parts. Graph capture runs your code once while recording every operation into a list, the intermediate representation (IR). A pass walks the IR and rewrites it; yours finds chains of element-wise ops and fuses them into one. Code generation turns each fused group back into code you can run. Take on arrays of numbers. Run op by op, it reads values from memory and writes . Fused, it reads and writes . In pure NumPy you get there by working in chunks small enough that the temporaries never leave the CPU cache.
Measure before you build. If element-wise ops take a fraction of the step and you make them times faster, the whole step speeds up by
which is Amdahl's law. If matrix multiplies take more than two thirds of the step, even infinite won't give 1.5×. Pick a model where element-wise work is at least a third of the step, such as a wide MLP with GELU trained with Adam.
DATA · your BL-11/BL-14 code
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- The fused training step is at least 1.5× faster than your unfused version, with identical outputs. Record it with `python tools/mlarena.py record BL-28 "<x>x, identical"`.
- Loss curves over 200 training steps match exactly between the fused and unfused runs.
- The README shows one captured graph, its fusion groups, and the generated source for one group.
Side quests
Terms in plain words
- Graph capture
- Running the code once with special objects that record every operation, so you get a list of ops you can inspect and rewrite.
- Intermediate representation (IR)
- The compiler's own description of the program, here a list of nodes with an op name, inputs and output shape.
- Op fusion
- Merging several operations into one, so intermediate arrays never get written to memory and read back.
- Element-wise op
- An operation where each output entry depends only on the same entry of the inputs, such as add, multiply or tanh.
- JIT compilation
- Turning code into machine code at run time, the first time it is called, as Numba's njit does.