Brief
A generative model learns what its training data looks like well enough to make new examples. You'll build two kinds, each small enough to train on a laptop, and use them to draw pixel-art sprites.
A variational autoencoder (VAE) has an encoder that maps an image to a distribution over a short latent code , and a decoder that maps a code back to an image. The encoder outputs a mean and a log-variance . To keep sampling differentiable, you draw with . This is the reparameterisation trick. The loss is reconstruction error plus a KL term that keeps the codes close to , so that afterwards you can sample a fresh code and decode a new sprite.
A diffusion model takes another route. It adds Gaussian noise to a training image until it's pure noise, and trains a network to predict the noise that was added. A closed form lets you jump straight to any step : , where shrinks from 1 towards 0 as grows. To generate, you start from noise and remove a little at each step. It's slower to sample than a VAE, but the pictures are usually sharper.
For data, use a free sprite pack. Kenney (kenney.nl) publishes CC0 packs such as Tiny Dungeon and 1-Bit Pack, and OpenGameArt.org hosts the Liberated Pixel Cup (LPC) character sprites, which need attribution. Slice the sheets into 16 × 16 or 32 × 32 tiles. A few hundred tiles is a small set, so stretch it with horizontal flips and colour shifts.
DATA · a public sprite set
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- Your VAE's validation reconstruction loss is within 5% of the reference VAE, or lower, recorded with `python tools/mlarena.py record BL-18 "<yours> vs <reference>"`.
- The README shows an 8 × 8 grid of VAE samples and an 8 × 8 grid of diffusion samples.
- The copy check shows new sprites, not near-copies of training tiles.
Side quests
Terms in plain words
- Generative model
- A model that learns what its training data looks like well enough to produce new examples of it.
- Latent code
- The short list of numbers a VAE squeezes each image into. Decoding a new code gives a new image.
- Reparameterisation trick
- Writing a random sample as mean plus spread times fixed noise, so gradients can flow through the mean and the spread.
- KL divergence
- A measure of how different two probability distributions are. In a VAE it keeps the codes close to a standard normal.
- Diffusion model
- A model trained to remove a little noise at a time. Starting from pure noise and repeating that step many times produces an image.