← All Base Form ML projects
BL-28NOT STARTEDBASE FORM · TRANSFORMERSMLPYTHON

Mini compiler

Record the operations your NumPy autograd runs, fuse the element-wise ones, and generate faster code.

EST.10 h
REWARD+100 XP
AFTERoriginal

ML-ARENA BENCHMARKS

YOUR NUMBER VS THE INDUSTRY'S
METRICINDUSTRYMINE
speed; output match
GOAL · identical outputs; ≥ 1.5× faster
your unfused version—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Your NumPy autograd from BL-11 runs one operation at a time. Each element-wise op reads its inputs from memory, writes a whole new array, and the next op reads that array back. For big arrays the time goes to moving data, not to maths. In BL-14 you saw the same thing from the other side: speed comes from how data moves. This project builds a tiny compiler that removes those round trips, the same idea behind torch.compile and XLA.

A compiler here has three parts. Graph capture runs your code once while recording every operation into a list, the intermediate representation (IR). A pass walks the IR and rewrites it; yours finds chains of element-wise ops and fuses them into one. Code generation turns each fused group back into code you can run. Take y=tanh⁡(a⋅b+c)⋅dy = \tanh(a \cdot b + c) \cdot d on arrays of NN numbers. Run op by op, it reads 7N7N values from memory and writes 4N4N. Fused, it reads 4N4N and writes NN. In pure NumPy you get there by working in chunks small enough that the temporaries never leave the CPU cache.

Measure before you build. If element-wise ops take a fraction ff of the step and you make them ss times faster, the whole step speeds up by

1(1−f)+f/s\frac{1}{(1 - f) + f / s}

which is Amdahl's law. If matrix multiplies take more than two thirds of the step, even infinite ss won't give 1.5×. Pick a model where element-wise work is at least a third of the step, such as a wide MLP with GELU trained with Adam.

VISUALFig. 1
UNFUSED: 4 PASSES, 3 TEMPORARIES t1 = a * b t2 = t1 + c t3 = tanh(t2) y = t3 * d MEMORY reads 7N values, writes 4N FUSED: 1 PASS for each i: y[i] = tanh(a[i] * b[i] + c[i]) * d[i] MEMORY reads 4N values, writes N read write
Unfused, every op writes a full temporary array to memory and the next op reads it back. Fused, one pass over memory: each chunk's temporaries stay in cache and only the result is written. Why it's here: This is the whole reason fusion pays off; it tells you which ops are worth grouping in objective 4.
HINT · OBJECTIVE 2
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 5
Stuck? Hints stay hidden until you ask.

DATA · your BL-11/BL-14 code

Objectives

0 / 6 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. The fused training step is at least 1.5× faster than your unfused version, with identical outputs. Record it with `python tools/mlarena.py record BL-28 "<x>x, identical"`.
  2. Loss curves over 200 training steps match exactly between the fused and unfused runs.
  3. The README shows one captured graph, its fusion groups, and the generated source for one group.

Side quests

Add a second backend that emits one loop per group and compiles it with numba.njit. Expect matches to about 1e-6, not bit for bit, since its tanh and exp can round differently.
Add common subexpression elimination and dead code removal passes, and count how many nodes each removes.

Terms in plain words

Graph capture
Running the code once with special objects that record every operation, so you get a list of ops you can inspect and rewrite.
Intermediate representation (IR)
The compiler's own description of the program, here a list of nodes with an op name, inputs and output shape.
Op fusion
Merging several operations into one, so intermediate arrays never get written to memory and read back.
Element-wise op
An operation where each output entry depends only on the same entry of the inputs, such as add, multiply or tanh.
JIT compilation
Turning code into machine code at run time, the first time it is called, as Numba's njit does.