← All Cracked ML projects
PL-I05NOT STARTEDINTERMEDIATEMLPYTHONIMPACT ★☆☆☆☆

word2vec SGNS vs gensim, then a contrastive rematch

Your skip-gram model must beat gensim on the Google analogy test. Then find out whether InfoNCE beats it.

EST.10 h
REWARD+300 XP
AFTERMikolov et al. + 1

ML-ARENA BENCHMARKS

PROOF: BENCH
METRICINDUSTRYMINE
accuracy on questions-words.txt (the Google analogy set, questions like king − man + woman ≈ queen) and training wall-clock, at dim=100, window=5, 5 epochs
GOAL · analogy accuracy ≥ gensim's at ≤ 3× its wall-clock; a written result on InfoNCE vs SGNS
gensim Word2Vec (same hyper-parameters) · measured locally—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

In PyTorch (no nn.Embedding optimiser shortcuts), implement skip-gram with negative sampling (SGNS): predict context words from a centre word, and train by telling real context words apart from random 'negative' words. Include frequent-word subsampling (randomly skip very common words like 'the'), a unigram^0.75 noise table (draw negatives in proportion to word frequency raised to the power 0.75, which gives rare words a bit more chance), a linear LR decay and sparse updates (only update the embedding rows that were used in the batch). Train on text8. Then swap in an in-batch InfoNCE loss (a contrastive loss where the other examples in the same batch act as the negatives) and compare at equal compute.

VICTORY CONDITIONS

  1. analogy accuracy ≥ gensim's at ≤ 3× its wall-clock; a written result on InfoNCE vs SGNS (accuracy on questions-words.txt (the Google analogy set, questions like king − man + woman ≈ queen) and training wall-clock, at dim=100, window=5, 5 epochs).
  2. A public repository with a README that shows your result and how to reproduce it.

Terms in plain words

SGNS (skip-gram with negative sampling)
word2vec's training method: score real (word, context) pairs high and random (word, noise) pairs low.
Subsampling
Randomly dropping very frequent words during training so they do not dominate the updates.
Unigram^0.75 noise table
A lookup table for drawing negative words with probability proportional to their frequency to the power 0.75.
InfoNCE
A contrastive loss that treats one matching pair as the right answer in a softmax over many non-matching pairs.
Analogy test
Checking whether vector arithmetic such as king − man + woman lands closest to queen.

Why it rates ★☆☆☆☆

word2vec is 2013 material. Only the controlled InfoNCE comparison adds anything.