Brief
In plain PyTorch (no peft/trl), write LoRA module injection with merge/unmerge. LoRA (low-rank adaptation) freezes the original weights and learns a small low-rank update beside each chosen layer; merging adds that update into the weights for fast inference, and unmerging takes it back out. Write an SFT loop (supervised fine-tuning on example responses). Then write the DPO loss with a frozen reference model and per-token log-prob gathering. DPO (direct preference optimisation) trains the model to give the preferred answer a higher probability than the rejected one, measured relative to the frozen reference model, without training a separate reward model. Apply it to Pythia-410M or Qwen-0.5B on Anthropic HH (a dataset of human preference pairs). Compare against TRL's DPOTrainer at identical hyper-parameters and seeds, and measure memory vs full fine-tuning.
Approach
LoRA replaces a frozen weight with , where starts random and starts at zero, so training begins from the exact base model. Merging computes once; unmerging subtracts it. Do the merge arithmetic in fp32 and cast back, or a merge-unmerge round trip will not return the original weights.
DPO needs four numbers per pair: the summed log-prob of the chosen answer and the rejected answer , under the policy and the frozen reference . The loss is
and the implicit reward of an answer is . With LoRA you do not need a second copy of the model: the reference is the same network with the adapters switched off, which is also what TRL does when you pass a PEFT model and no reference. Run the reference pass under torch.no_grad().
Order: LoRA module with its three exactness tests, then data and masks, then SFT, then the DPO loss, then the TRL comparison. To match TRL, read the DPOConfig of the version you install and copy every value, because defaults change between releases (in recent versions β is 0.1 and the loss type is sigmoid, which is the formula above). Pythia-410M or Qwen-0.5B fits comfortably on a single 24 GB GPU for both LoRA and full fine-tuning at moderate sequence lengths. Budget a few GPU-hours per run and about five runs (SFT, your DPO, TRL DPO, full fine-tune for memory, generation for the win rate).
Where people get stuck
- Summing over the prompt. The prompt is shared, so its log-prob cancels in exact arithmetic but not in practice with truncation. Mask it out.
- Mean instead of sum. DPO uses the summed log-prob of the response. Averaging per token changes the loss and you will not match TRL.
- Left truncation and padding. HH conversations are long. If you truncate differently from TRL, the curves will differ for reasons that have nothing to do with the loss.
- Unfair memory comparison. Peak memory is dominated by activations at long sequence lengths. Use the same batch, length, dtype and gradient checkpointing in both runs, or the 40% target is meaningless.
- Rising margins with falling quality. DPO can push both answers' log-probs down while the margin grows. Log the chosen and rejected log-probs separately, not just the margin.
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- Held-out preference accuracy within 1 point of TRL's DPOTrainer at identical settings. Record it with `python tools/mlarena.py record PL-A02 "<yours> vs <TRL>"`.
- LoRA r = 16 peak GPU memory is at most 40% of a full fine-tune at the same batch size and sequence length.
- DPO beats SFT in at least 60% of 500 held-out prompts under a fixed reward model, with a 95% interval reported.
Side quests
Terms in plain words
- LoRA
- Fine-tuning by learning two small matrices whose product is added to a frozen weight matrix, so only a few parameters train.
- Merge / unmerge
- Adding the LoRA update into the base weights for inference, or subtracting it back out to recover the original model.
- SFT (supervised fine-tuning)
- Ordinary next-token training on example prompts and good responses.
- DPO
- A fine-tuning loss that pushes up the chosen answer and pushes down the rejected one, compared with a frozen copy of the starting model.
- Reference model
- A frozen copy of the model before DPO, used so the trained model does not drift too far.
- Implicit reward margin
- How much more the trained model prefers the chosen answer over the rejected one, compared with the reference model.
- Loss mask
- A 0/1 vector that marks which tokens count toward the loss, here only the response tokens and never the prompt.
Why it rates ★★☆☆☆
Post-training literacy is valued, but matching TRL on a small model is now common.