Brief
Word2vec turns each word into a list of a few hundred numbers, called an embedding. Words used in similar contexts end up close together. Nobody labels anything: the only training signal is which words appear near which. The famous result is that directions carry meaning, so the vector for king − man + woman lands near queen.
First, text has to become tokens, and tokenisation decides what the units are. Word-level tokens are the simplest: split on spaces, keep words seen at least 5 times, and map the rest to one unknown token. Byte-pair encoding (BPE) starts from single characters and keeps merging the most frequent neighbouring pair, so a rare word splits into common pieces. Modern language models use BPE. Word2vec uses whole words, and objective 1 shows you why.
You'll write skip-gram with negative sampling. Each word has a centre vector and a context vector . For a centre word , a real context word and random noise words, the loss is
where is the sigmoid. Real pairs get pulled together and noise pairs get pushed apart.
Use text8: the first 100 MB of a cleaned English Wikipedia dump, already lowercase and space-separated. Matt
Mahoney prepared it, and the gensim-data downloader serves it as text8. The analogy test set is
questions-words.txt from the original word2vec release. gensim ships a copy in its test data, so
gensim.test.utils.datapath("questions-words.txt") gives you its path.
DATA · a Wikipedia text slice
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- Your analogy accuracy is within 5 points of gensim on the same corpus and settings, recorded with `python tools/mlarena.py record BL-16 "<yours>% vs <gensim>%"`.
- Both models answer the same questions. Skip any question with a word outside the vocabulary, as gensim does.
- The README lists nearest neighbours for five words and splits the score into the semantic and syntactic sections.
Side quests
Terms in plain words
- Embedding
- A learned list of numbers that stands for a token. Tokens used in similar ways end up with similar lists.
- Tokenisation
- Splitting raw text into the units a model works with, such as words, pieces of words or single characters.
- Byte-pair encoding (BPE)
- A tokeniser that starts from single characters and keeps merging the most frequent neighbouring pair into a new token.
- Skip-gram
- The word2vec task of predicting the words around a centre word from that word's vector.
- Negative sampling
- Training on each real word pair alongside a few random fake pairs, so the model only has to tell real from fake.