← All Cracked ML projects
PL-A11NOT STARTEDADVANCEDMLPYTHONIMPACT ★★★☆☆

Triton kernels on the roofline, up to FlashAttention

Write fused softmax, LayerNorm and a FlashAttention forward/backward in Triton, and place each kernel on a roofline plot.

EST.35 h
REWARD+400 XP
AFTERTriton tutorial + 2

ML-ARENA BENCHMARKS

PROOF: BENCH
METRICINDUSTRYMINE
attention TFLOP/s at sequence length 4k, head-dim 64, fp16 (forward and backward); LayerNorm's % of peak memory bandwidth
GOAL · attention forward ≥ 80% and backward ≥ 60% of SDPA's TFLOP/s; LayerNorm ≥ 85% of peak bandwidth; outputs match SDPA to 1e-3
PyTorch scaled_dot_product_attention (SDPA, flash backend) and the Triton fused-attention tutorial · measured locally—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Write Triton kernels (Triton is a Python DSL for writing GPU kernels; CUDA C is left to the C tracks) for fused softmax, LayerNorm forward/backward, a tiled fp16 matmul, and FlashAttention-2-style forward and backward with causal masking. FlashAttention computes attention block by block in fast on-chip memory, so the full sequence-by-sequence attention matrix is never written to GPU memory. Each kernel gets a roofline analysis: arithmetic intensity (FLOPs done per byte moved), achieved FLOP/s and bandwidth vs the GPU's peak, with autotuned block sizes. The roofline plot shows whether each kernel is limited by compute or by memory bandwidth.

VICTORY CONDITIONS

  1. attention forward ≥ 80% and backward ≥ 60% of SDPA's TFLOP/s; LayerNorm ≥ 85% of peak bandwidth; outputs match SDPA to 1e-3 (attention TFLOP/s at sequence length 4k, head-dim 64, fp16 (forward and backward); LayerNorm's % of peak memory bandwidth).
  2. A public repository with a README that shows your result and how to reproduce it.

Terms in plain words

Kernel
A single function that runs in parallel on the GPU, such as one softmax or one matmul.
Triton
A Python-based language for writing GPU kernels that handles many low-level details for you.
Fused kernel
Several operations done in one kernel so intermediate results stay in fast memory instead of round-tripping to GPU memory.
FlashAttention
An exact attention algorithm that works on tiles of queries and keys so it never stores the full attention matrix.
Roofline
A plot of achievable speed against arithmetic intensity, showing a kernel's ceiling from memory bandwidth or from compute.
Arithmetic intensity
How many floating-point operations a kernel does for each byte it reads or writes.

Why it rates ★★★☆☆

Kernel skill that holds up on a roofline is scarce and directly hireable.