Brief
You have a folder of photos and you want to type "dog on a beach" and see the right ones. Nobody labelled them. The trick, from OpenAI's CLIP, is to train an image encoder and a text encoder so that a picture and a sentence about it end up as nearby vectors in one shared space. Search then works like BL-25: embed the query, find the nearest images.
The image side is a vision transformer. It treats an image as a sequence: a 224 × 224 picture cut into 16 × 16 patches gives 196 patches, each flattened and linearly projected into a token. From there it is the same transformer you built in BL-21, without the causal mask, because every patch may look at every other patch.
The two encoders learn together through a contrastive loss. In a batch of pairs, normalise the image embeddings and text embeddings , and compare every image with every caption:
Row of this matrix is a classification problem whose right answer is column , and the same holds for each column. The loss is the average of cross-entropy over the rows and over the columns. is the temperature, learned with the rest.
CLIP saw 400 million pairs. You have tens of thousands, so you won't train encoders from scratch. You'll freeze a pretrained image model and a pretrained text model and learn only the projections into a shared space. That is the realistic way to get close to CLIP with this much data.
DATA · a public image–caption set
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- Your model's text-to-image recall@5 on the test split is within 10% of pretrained CLIP's on the same split. Record it with `python tools/mlarena.py record BL-26 "<yours> vs <clip> R@5"`.
- No test image, or any caption of one, was used in training.
- Your patch-embedding check passes, and the README shows the top 5 results for five queries on your own photos.
Side quests
Terms in plain words
- Vision transformer (ViT)
- A transformer for images. It cuts the picture into fixed-size patches and treats each patch as one token.
- CLIP
- A model trained on image and caption pairs so that a picture and a sentence describing it land close together in one shared space.
- Contrastive loss
- A loss that pulls each matching image and caption together and pushes every non-matching pair in the batch apart.
- Temperature
- A number that divides the similarities before the softmax. Smaller values make the model more decisive; CLIP learns it.
- Zero-shot
- Using a pretrained model on a new task without training it on that task's data.