Brief
In PyTorch (no nn.Embedding optimiser shortcuts), implement skip-gram with negative sampling (SGNS): predict context words from a centre word, and train by telling real context words apart from random 'negative' words. Include frequent-word subsampling (randomly skip very common words like 'the'), a unigram^0.75 noise table (draw negatives in proportion to word frequency raised to the power 0.75, which gives rare words a bit more chance), a linear LR decay and sparse updates (only update the embedding rows that were used in the batch). Train on text8. Then swap in an in-batch InfoNCE loss (a contrastive loss where the other examples in the same batch act as the negatives) and compare at equal compute.
VICTORY CONDITIONS
- analogy accuracy ≥ gensim's at ≤ 3× its wall-clock; a written result on InfoNCE vs SGNS (accuracy on questions-words.txt (the Google analogy set, questions like king − man + woman ≈ queen) and training wall-clock, at dim=100, window=5, 5 epochs).
- A public repository with a README that shows your result and how to reproduce it.
Terms in plain words
- SGNS (skip-gram with negative sampling)
- word2vec's training method: score real (word, context) pairs high and random (word, noise) pairs low.
- Subsampling
- Randomly dropping very frequent words during training so they do not dominate the updates.
- Unigram^0.75 noise table
- A lookup table for drawing negative words with probability proportional to their frequency to the power 0.75.
- InfoNCE
- A contrastive loss that treats one matching pair as the right answer in a softmax over many non-matching pairs.
- Analogy test
- Checking whether vector arithmetic such as king − man + woman lands closest to queen.
Why it rates ★☆☆☆☆
word2vec is 2013 material. Only the controlled InfoNCE comparison adds anything.