Brief
Search by keyword fails when you remember the idea but not the words. This project builds a question-answering tool over your own notes that finds passages by meaning, shows its sources, and is measured against a strong keyword baseline. The pattern is retrieval-augmented generation, or RAG: retrieve first, then let a language model answer from what was retrieved.
Retrieval here has three stages. BM25 is the keyword baseline, and it is better than it sounds. Dense retrieval uses the embeddings you met in BL-16, now for whole passages: a passage and a question become unit vectors and the score is . That is a nearest-neighbour search, the same idea as k-NN in BL-04. The two methods fail on different questions, so you combine their ranked lists with reciprocal rank fusion:
Finally a reranker reads the question and each candidate together, which is slower but much sharper, so you only run it on the top 30.
Evaluation comes first. You learned in BL-24 that a test set you tuned on stops being a test. Write the 50 questions and their answers before building anything, and don't edit them after you see results.
DATA · your notes
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- Your best pipeline's recall@5 on the 50 frozen questions is at least 10 points above BM25. Record it with `python tools/mlarena.py record BL-25 "<yours> vs <bm25> recall@5"`.
- A table in the README shows recall@5 and MRR for BM25, dense, fused and reranked retrieval.
- In 10 hand-checked answers, every citation points to a chunk that actually supports the answer.
Side quests
Terms in plain words
- Retrieval-augmented generation (RAG)
- Find the passages most likely to hold the answer, then give them to a language model to answer from, with citations.
- BM25
- A classic keyword ranking that scores a passage by how often it contains the query's words, weighting rare words more.
- Dense retrieval
- Embed the question and every passage as vectors and return the passages whose vectors are closest by cosine similarity.
- Reranker
- A slower model that reads the question and one passage together and scores how well they match. It reorders a short list.
- recall@5
- The fraction of questions where at least one correct passage appears in the top 5 results.