Brief
Implement a ViT-B/32 image tower (a vision transformer that cuts the image into 32×32-pixel patches and treats each patch as a token) and a text transformer in PyTorch. Load the OpenAI CLIP weights into your modules and reproduce zero-shot CIFAR-10 and ImageNet-V2 accuracy with prompt ensembling. Zero-shot means you classify by comparing the image embedding with text embeddings of the class names, with no training on that dataset. Prompt ensembling averages the text embeddings from several templates, such as 'a photo of a {class}'. Then train a small CLIP from scratch on COCO captions and compare InfoNCE (a softmax over all image-text pairings in the batch) vs the SigLIP sigmoid loss (an independent match / no-match score for each pair) at batch 256.
VICTORY CONDITIONS
- embedding cosine ≥ 0.9999 and zero-shot within 0.1 pt of the reference; a controlled InfoNCE vs SigLIP result at equal compute (cosine similarity between your embeddings and the reference embeddings; zero-shot top-1 (CIFAR-10, ImageNet-V2); zero-shot accuracy of your from-scratch model).
- A public repository with a README that shows your result and how to reproduce it.
Terms in plain words
- CLIP
- A pair of encoders, one for images and one for text, trained so matching image-caption pairs end up with similar embeddings.
- ViT-B/32
- A base-size vision transformer that splits each image into 32×32-pixel patches.
- Zero-shot classification
- Classifying images by picking the class name whose text embedding is closest, without training on that dataset.
- Prompt ensembling
- Averaging the embeddings of several phrasings of each class name to get a more robust class vector.
- SigLIP loss
- A CLIP loss that scores each image-text pair separately with a sigmoid, instead of a softmax over the whole batch.
Why it rates ★★☆☆☆
Porting weights proves correctness and the loss comparison adds some insight, but small-scale CLIP is well studied.