← All Cracked ML projects
PL-A02NOT STARTEDADVANCEDMLPYTHONIMPACT ★★☆☆☆

LoRA + DPO from scratch, tested against TRL

Write the 30 lines that matter in DPO and LoRA yourself, and get the same curves as the library everyone uses.

EST.24 h
REWARD+400 XP
AFTERRafailov et al. + 2

ML-ARENA BENCHMARKS

PROOF: BENCH
METRICINDUSTRYMINE
held-out preference accuracy (how often the implicit reward margin, the model's preference for the chosen over the rejected answer, is > 0); win rate vs SFT under a fixed reward model; peak GPU memory
GOAL · preference accuracy within 1 pt of TRL; LoRA r=16 (rank 16) peak memory ≤ 40% of a full fine-tune; win rate vs SFT ≥ 60%
TRL DPOTrainer + PEFT LoRA (same config) · measured locally—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

In plain PyTorch (no peft/trl), write LoRA module injection with merge/unmerge. LoRA (low-rank adaptation) freezes the original weights and learns a small low-rank update beside each chosen layer; merging adds that update into the weights for fast inference, and unmerging takes it back out. Write an SFT loop (supervised fine-tuning on example responses). Then write the DPO loss with a frozen reference model and per-token log-prob gathering. DPO (direct preference optimisation) trains the model to give the preferred answer a higher probability than the rejected one, measured relative to the frozen reference model, without training a separate reward model. Apply it to Pythia-410M or Qwen-0.5B on Anthropic HH (a dataset of human preference pairs). Compare against TRL's DPOTrainer at identical hyper-parameters and seeds, and measure memory vs full fine-tuning.

Approach

LoRA replaces a frozen weight W0W_0 with W0+αrBAW_0 + \frac{\alpha}{r}BA, where A∈Rr×dinA \in \mathbb{R}^{r \times d_\text{in}} starts random and B∈Rdout×rB \in \mathbb{R}^{d_\text{out} \times r} starts at zero, so training begins from the exact base model. Merging computes W0+αrBAW_0 + \frac{\alpha}{r}BA once; unmerging subtracts it. Do the merge arithmetic in fp32 and cast back, or a merge-unmerge round trip will not return the original weights.

DPO needs four numbers per pair: the summed log-prob of the chosen answer ywy_w and the rejected answer yly_l, under the policy πθ\pi_\theta and the frozen reference πref\pi_\text{ref}. The loss is

L=−log⁡σ ⁣(β[(log⁡πθ(yw∣x)−log⁡πref(yw∣x))−(log⁡πθ(yl∣x)−log⁡πref(yl∣x))])\mathcal{L} = -\log \sigma\!\Big(\beta\big[(\log \pi_\theta(y_w|x) - \log \pi_\text{ref}(y_w|x)) - (\log \pi_\theta(y_l|x) - \log \pi_\text{ref}(y_l|x))\big]\Big)

and the implicit reward of an answer is r^=β(log⁡πθ−log⁡πref)\hat r = \beta(\log \pi_\theta - \log \pi_\text{ref}). With LoRA you do not need a second copy of the model: the reference is the same network with the adapters switched off, which is also what TRL does when you pass a PEFT model and no reference. Run the reference pass under torch.no_grad().

Order: LoRA module with its three exactness tests, then data and masks, then SFT, then the DPO loss, then the TRL comparison. To match TRL, read the DPOConfig of the version you install and copy every value, because defaults change between releases (in recent versions β is 0.1 and the loss type is sigmoid, which is the formula above). Pythia-410M or Qwen-0.5B fits comfortably on a single 24 GB GPU for both LoRA and full fine-tuning at moderate sequence lengths. Budget a few GPU-hours per run and about five runs (SFT, your DPO, TRL DPO, full fine-tune for memory, generation for the win rate).

OVERVIEWFig. 1
ONE PREFERENCE PAIR prompt xy_w prompt xy_l both answers go throughgrey tokens: loss mask 0 both modelsyellow tokens: summed TRAINABLE POLICY π_θ W₀ + (α/r)·B·A only A, B get gradients FROZEN REFERENCE π_ref W₀ (LoRA disabled) no grad, same tokens log π_θ(y_w|x) log π_θ(y_l|x) log π_ref(y_w|x) log π_ref(y_l|x) MARGIN m (θ_w − ref_w) − (θ_l − ref_l) LOSS −log σ(β·m) TWO CHECKS BEFORE ANY TRAINING 1. At step 0 policy and reference are identical, so m = 0 and the loss is exactly ln 2 ≈ 0.693. 2. Each log π(y|x) sums over answer tokens only, gathered at the target ids shifted one step left.
What one DPO step computes: two answers, two models (one of them is the same weights with LoRA off), four summed log-probs, one margin. Why it's here: Most DPO bugs are a wrong mask or a wrong sum in one of these four boxes. Check each box separately.

Where people get stuck

  • Summing over the prompt. The prompt is shared, so its log-prob cancels in exact arithmetic but not in practice with truncation. Mask it out.
  • Mean instead of sum. DPO uses the summed log-prob of the response. Averaging per token changes the loss and you will not match TRL.
  • Left truncation and padding. HH conversations are long. If you truncate differently from TRL, the curves will differ for reasons that have nothing to do with the loss.
  • Unfair memory comparison. Peak memory is dominated by activations at long sequence lengths. Use the same batch, length, dtype and gradient checkpointing in both runs, or the 40% target is meaningless.
  • Rising margins with falling quality. DPO can push both answers' log-probs down while the margin grows. Log the chosen and rejected log-probs separately, not just the margin.
HINT · OBJECTIVE 2
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 4
Stuck? Hints stay hidden until you ask.

Objectives

0 / 7 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. Held-out preference accuracy within 1 point of TRL's DPOTrainer at identical settings. Record it with `python tools/mlarena.py record PL-A02 "<yours> vs <TRL>"`.
  2. LoRA r = 16 peak GPU memory is at most 40% of a full fine-tune at the same batch size and sequence length.
  3. DPO beats SFT in at least 60% of 500 held-out prompts under a fixed reward model, with a 95% interval reported.

Side quests

Swap the loss for IPO or a length-normalised DPO and check whether the chosen-response log-prob still falls during training, a known DPO side effect.
Sweep β over 0.01, 0.1 and 0.5 and plot win rate against KL from the reference model.

Terms in plain words

LoRA
Fine-tuning by learning two small matrices whose product is added to a frozen weight matrix, so only a few parameters train.
Merge / unmerge
Adding the LoRA update into the base weights for inference, or subtracting it back out to recover the original model.
SFT (supervised fine-tuning)
Ordinary next-token training on example prompts and good responses.
DPO
A fine-tuning loss that pushes up the chosen answer and pushes down the rejected one, compared with a frozen copy of the starting model.
Reference model
A frozen copy of the model before DPO, used so the trained model does not drift too far.
Implicit reward margin
How much more the trained model prefers the chosen answer over the rejected one, compared with the reference model.
Loss mask
A 0/1 vector that marks which tokens count toward the loss, here only the response tokens and never the prompt.

Why it rates ★★☆☆☆

Post-training literacy is valued, but matching TRL on a small model is now common.