Brief
Build a multinomial naive Bayes classifier over character 1–5-grams (every run of 1 to 5 characters in the text). Use feature hashing (map each n-gram into a fixed number of buckets with a hash function instead of storing a vocabulary), n-gram pruning (drop rare or uninformative n-grams) and int8-quantised log-probabilities (store each log-probability as an 8-bit integer instead of a 32-bit float). Together these let the whole model ship in ≤ 100 KB. Evaluate on the WiLI-2018 languages that fastText's lid.176 model also covers. The size budget forces real feature-engineering choices.
VICTORY CONDITIONS
- within 2 accuracy points of lid.176.ftz with a model ≤ 100 KB (macro accuracy (accuracy averaged per language, so each language counts equally) on the WiLI-2018 test paragraphs, for the languages that overlap with lid.176, at a given model size).
- A public repository with a README that shows your result and how to reproduce it.
Terms in plain words
- Character n-gram
- A run of n consecutive characters, such as 'th' or 'ing', used as a feature.
- Feature hashing
- Mapping each feature to a bucket number with a hash function, so you never store a vocabulary and memory stays fixed.
- int8 quantisation
- Storing numbers as 8-bit integers plus a scale factor, which is 4 times smaller than 32-bit floats.
- Macro accuracy
- Accuracy computed separately for each class and then averaged, so rare languages count as much as common ones.
Why it rates ★★☆☆☆
A tiny model that stays competitive under a size budget is a nice engineering story, though language identification is a mature problem.