← All Base Form ML projects
BL-02NOT STARTEDBASE FORM · CLASSICAL MLMLPYTHON

SMS spam filter

A spam filter you can paste messages into, built from word counts and Bayes' rule.

EST.5 h
REWARD+100 XP
AFTERoriginal

ML-ARENA BENCHMARKS

YOUR NUMBER VS THE INDUSTRY'S
METRICINDUSTRYMINE
F1 on held-out messages
GOAL · within 0.5%
sklearn MultinomialNB—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Spam filters were one of the first places machine learning went into daily use, and naive Bayes was the method that did it. You'll build one for text messages. The finished tool reads a message you paste in, says spam or ham (real message), and shows which words decided it.

The data is the SMS Spam Collection from the UCI Machine Learning Repository: 5,574 English text messages, each labelled ham or spam. About 13% are spam. That imbalance matters. A filter that calls everything ham is 87% accurate and useless, which is why you score with F1 on the spam class instead of accuracy.

The core idea is Bayes' rule. You want P(spam∣message)P(\text{spam} \mid \text{message}), but it's easier to learn P(message∣spam)P(\text{message} \mid \text{spam}) from counts. Naive Bayes makes one bold guess: given the class, words appear independently. That turns the message probability into a product of word probabilities, and taking logs turns the product into a sum:

log⁡P(c∣x)=log⁡P(c)+∑wnwlog⁡P(w∣c)+const,\log P(c \mid x) = \log P(c) + \sum_{w} n_w \log P(w \mid c) + \text{const},

where nwn_w is how often word ww appears in the message. Each word probability is a smoothed count, P(w∣c)=Nwc+αNc+α∣V∣P(w \mid c) = \frac{N_{wc} + \alpha}{N_c + \alpha \lvert V \rvert}, where NwcN_{wc} is how often ww appears in class cc, NcN_c is the total word count of class cc, ∣V∣\lvert V \rvert is the vocabulary size and α=1\alpha = 1. Pick whichever class has the larger score.

VISUALFig. 1
TERM ← HAM SPAM → prior "free" "prize" "call" "now" total spam illustrative log-odds, one bar per word
Each word adds or subtracts log-odds. The prior starts the total, and the sign of the final sum is the decision. Why it's here: Shows why the log form is a sum you can read word by word, which is what the paste-in tool in objective 6 prints.
HINT · OBJECTIVE 3
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 5
Stuck? Hints stay hidden until you ask.

DATA · UCI SMS Spam Collection

Objectives

0 / 6 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. Your F1 on the held-out spam messages is within 0.5% of sklearn MultinomialNB on the same split and vocabulary. Record it with `python tools/mlarena.py record BL-02 "<your F1>"`.
  2. The vocabulary and probabilities come only from training messages, and every probability is handled in log space.
  3. The paste-in tool works on new text you type, including words it has never seen.

Side quests

Try bigrams (pairs of words) as extra features and see whether F1 improves.
Plot precision and recall as you move the decision threshold, and pick the threshold you would ship if a missed real message costs ten times a missed spam.

Terms in plain words

Naive Bayes
A classifier that applies Bayes' rule while pretending every word appears independently of the others, given the class.
Generative classifier
A model that learns what each class looks like, P(words | class), and then uses Bayes' rule to work out the class.
Bag of words
A message turned into a count for each word in the vocabulary, ignoring word order.
Laplace smoothing
Adding a small count (usually 1) to every word, so a word never seen in spam doesn't make spam impossible.
F1 score
The harmonic mean of precision and recall for the spam class. It stays low if the filter misses spam or flags real messages.