Brief
Common Crawl is a public crawl of the web that also publishes a graph of which domains link to which. Stream its latest domain-level web graph (cc-main-2026-jul-aug-sep: 133.2M nodes, 2.1B edges) into a memory-mapped CSR you build yourself (a sparse matrix stored on disk that the operating system pages into RAM as needed). Run PageRank out of core (with data larger than RAM) in NumPy on a machine with <= 32 GB RAM, in three variants: power iteration, Gauss-Seidel (uses each updated value immediately, which often converges faster) and extrapolation-accelerated PageRank (uses the last few iterates to estimate where the sequence is heading and jumps there). Then compare with Common Crawl's published domain ranks.
VICTORY CONDITIONS
- rho >= 0.999 on the top 1M; full run < 3 h with peak RSS < 32 GB; extrapolation cuts iterations by >= 30% vs plain power iteration (Spearman rho (a rank correlation where 1 means identical order) vs Common Crawl's published domain PageRank (top 1M); wall time; peak RSS (the most RAM the process used at any moment)).
- A public repository with a README that shows your result and how to reproduce it.
Terms in plain words
- Memory map (memmap)
- A file on disk that a program can index like an array, with the operating system loading pieces into RAM on demand.
- Out of core
- Processing data that is larger than RAM by streaming it from disk in pieces.
- Gauss-Seidel iteration
- An iterative method that uses each newly updated value immediately within the same sweep, instead of waiting for the next iteration.
- Extrapolation acceleration
- Combining several recent iterates to cancel the slowest-decaying error component and reach the answer in fewer iterations.
- Spearman rho
- The correlation between two rankings, computed on rank positions rather than raw values.
Why it rates ★★★☆☆
Impressive single-machine scale engineering, but reproducing an existing ranking is not research.