← All Base Form ML projects
BL-16NOT STARTEDBASE FORM · DEEP LEARNINGMLPYTHON

Word map

Train word2vec on Wikipedia text and solve analogies like king − man + woman.

EST.8 h
REWARD+100 XP
AFTERMikolov et al.

ML-ARENA BENCHMARKS

YOUR NUMBER VS THE INDUSTRY'S
METRICINDUSTRYMINE
analogy accuracy
GOAL · within 5 points
gensim Word2Vec, same corpus—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Word2vec turns each word into a list of a few hundred numbers, called an embedding. Words used in similar contexts end up close together. Nobody labels anything: the only training signal is which words appear near which. The famous result is that directions carry meaning, so the vector for king − man + woman lands near queen.

First, text has to become tokens, and tokenisation decides what the units are. Word-level tokens are the simplest: split on spaces, keep words seen at least 5 times, and map the rest to one unknown token. Byte-pair encoding (BPE) starts from single characters and keeps merging the most frequent neighbouring pair, so a rare word splits into common pieces. Modern language models use BPE. Word2vec uses whole words, and objective 1 shows you why.

You'll write skip-gram with negative sampling. Each word has a centre vector vv and a context vector uu. For a centre word cc, a real context word oo and KK random noise words, the loss is

L=−log⁡σ(uo⊤vc)−∑k=1Klog⁡σ(−uk⊤vc)L = -\log \sigma(u_o^\top v_c) - \sum_{k=1}^{K} \log \sigma(-u_k^\top v_c)

where σ\sigma is the sigmoid. Real pairs get pulled together and noise pairs get pushed apart.

Use text8: the first 100 MB of a cleaned English Wikipedia dump, already lowercase and space-separated. Matt Mahoney prepared it, and the gensim-data downloader serves it as text8. The analogy test set is questions-words.txt from the original word2vec release. gensim ships a copy in its test data, so gensim.test.utils.datapath("questions-words.txt") gives you its path.

VISUALFig. 1
man woman king queen woman − man same step king − man + woman lands near queen 2D sketch of high-dimensional vectors
Analogies as vector offsets, drawn in 2D for illustration. The step from man to woman is roughly the same as the step from king to queen. Why it's here: Objective 5 turns this picture into code: start at king, add the man-to-woman step, and look for the nearest word.
HINT · OBJECTIVE 3
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 5
Stuck? Hints stay hidden until you ask.

DATA · a Wikipedia text slice

Objectives

0 / 6 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. Your analogy accuracy is within 5 points of gensim on the same corpus and settings, recorded with `python tools/mlarena.py record BL-16 "<yours>% vs <gensim>%"`.
  2. Both models answer the same questions. Skip any question with a word outside the vocabulary, as gensim does.
  3. The README lists nearest neighbours for five words and splits the score into the semantic and syntactic sections.

Side quests

Project countries and their capitals to 2D with PCA and plot the arrows between them.
Build word vectors as the sum of BPE-piece vectors, in the style of fastText, and test them on rare words.

Terms in plain words

Embedding
A learned list of numbers that stands for a token. Tokens used in similar ways end up with similar lists.
Tokenisation
Splitting raw text into the units a model works with, such as words, pieces of words or single characters.
Byte-pair encoding (BPE)
A tokeniser that starts from single characters and keeps merging the most frequent neighbouring pair into a new token.
Skip-gram
The word2vec task of predicting the words around a centre word from that word's vector.
Negative sampling
Training on each real word pair alongside a few random fake pairs, so the model only has to tell real from fake.