Brief
Write Triton kernels (Triton is a Python DSL for writing GPU kernels; CUDA C is left to the C tracks) for fused softmax, LayerNorm forward/backward, a tiled fp16 matmul, and FlashAttention-2-style forward and backward with causal masking. FlashAttention computes attention block by block in fast on-chip memory, so the full sequence-by-sequence attention matrix is never written to GPU memory. Each kernel gets a roofline analysis: arithmetic intensity (FLOPs done per byte moved), achieved FLOP/s and bandwidth vs the GPU's peak, with autotuned block sizes. The roofline plot shows whether each kernel is limited by compute or by memory bandwidth.
VICTORY CONDITIONS
- attention forward ≥ 80% and backward ≥ 60% of SDPA's TFLOP/s; LayerNorm ≥ 85% of peak bandwidth; outputs match SDPA to 1e-3 (attention TFLOP/s at sequence length 4k, head-dim 64, fp16 (forward and backward); LayerNorm's % of peak memory bandwidth).
- A public repository with a README that shows your result and how to reproduce it.
Terms in plain words
- Kernel
- A single function that runs in parallel on the GPU, such as one softmax or one matmul.
- Triton
- A Python-based language for writing GPU kernels that handles many low-level details for you.
- Fused kernel
- Several operations done in one kernel so intermediate results stay in fast memory instead of round-tripping to GPU memory.
- FlashAttention
- An exact attention algorithm that works on tiles of queries and keys so it never stores the full attention matrix.
- Roofline
- A plot of achievable speed against arithmetic intensity, showing a kernel's ceiling from memory bandwidth or from compute.
- Arithmetic intensity
- How many floating-point operations a kernel does for each byte it reads or writes.
Why it rates ★★★☆☆
Kernel skill that holds up on a roofline is scarce and directly hireable.