← All Base Form ML projects
BL-26NOT STARTEDBASE FORM · TRANSFORMERSMLPYTHON

Search photos by text

Type 'dog on a beach' and find matching photos from your own collection with CLIP-style image and text embeddings.

EST.9 h
REWARD+100 XP
AFTERoriginal

ML-ARENA BENCHMARKS

YOUR NUMBER VS THE INDUSTRY'S
METRICINDUSTRYMINE
recall@5
GOAL · within 10% of pretrained CLIP
pretrained CLIP zero-shot · measured—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

You have a folder of photos and you want to type "dog on a beach" and see the right ones. Nobody labelled them. The trick, from OpenAI's CLIP, is to train an image encoder and a text encoder so that a picture and a sentence about it end up as nearby vectors in one shared space. Search then works like BL-25: embed the query, find the nearest images.

The image side is a vision transformer. It treats an image as a sequence: a 224 × 224 picture cut into 16 × 16 patches gives 196 patches, each flattened and linearly projected into a token. From there it is the same transformer you built in BL-21, without the causal mask, because every patch may look at every other patch.

The two encoders learn together through a contrastive loss. In a batch of NN pairs, normalise the image embeddings uiu_i and text embeddings vjv_j, and compare every image with every caption:

sij=ui⋅vjτs_{ij} = \frac{u_i \cdot v_j}{\tau}

Row ii of this N×NN \times N matrix is a classification problem whose right answer is column ii, and the same holds for each column. The loss is the average of cross-entropy over the rows and over the columns. τ\tau is the temperature, learned with the rest.

CLIP saw 400 million pairs. You have tens of thousands, so you won't train encoders from scratch. You'll freeze a pretrained image model and a pretrained text model and learn only the projections into a shared space. That is the realistic way to get close to CLIP with this much data.

VISUALFig. 1
CAPTIONS (TEXT EMBEDDINGS v) IMAGES (u) v1v2v3v4v5 u1 u2 u3 u4 u5 ↑↑↑↑↑ matching pair push the score u·v up every other pair push the score down one row softmax over captions; the right answer for image 3 is caption 3 Columns get the same treatment, so the loss averages both directions.
One batch: every image is scored against every caption. Matching pairs sit on the diagonal. Why it's here: The loss in objective 5 is cross-entropy along each row and each column of this matrix; seeing it makes the code short.
HINT · OBJECTIVE 5
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 3
Stuck? Hints stay hidden until you ask.

DATA · a public image–caption set

Objectives

0 / 6 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. Your model's text-to-image recall@5 on the test split is within 10% of pretrained CLIP's on the same split. Record it with `python tools/mlarena.py record BL-26 "<yours> vs <clip> R@5"`.
  2. No test image, or any caption of one, was used in training.
  3. Your patch-embedding check passes, and the README shows the top 5 results for five queries on your own photos.

Side quests

Unfreeze the last block of each encoder and see how much recall@5 moves.
Use the trained model as a zero-shot classifier by embedding "a photo of a {label}" for each class name.

Terms in plain words

Vision transformer (ViT)
A transformer for images. It cuts the picture into fixed-size patches and treats each patch as one token.
CLIP
A model trained on image and caption pairs so that a picture and a sentence describing it land close together in one shared space.
Contrastive loss
A loss that pulls each matching image and caption together and pushes every non-matching pair in the batch apart.
Temperature
A number that divides the similarities before the softmax. Smaller values make the model more decisive; CLIP learns it.
Zero-shot
Using a pretrained model on a new task without training it on that task's data.