← All Cracked Math projects
PM-W02NOT STARTEDWOWMATHPYTHONIMPACT ★★★☆☆

PageRank the web on one machine: Common Crawl's domain graph

133 million domains, 2.1 billion links, one laptop, NumPy memmaps, and Common Crawl's own ranks as the answer key.

EST.45 h
REWARD+600 XP
AFTERCommon Crawl web graphs + 1

ML-ARENA BENCHMARKS

PROOF: BENCH
METRICINDUSTRYMINE
Spearman rho (a rank correlation where 1 means identical order) vs Common Crawl's published domain PageRank (top 1M); wall time; peak RSS (the most RAM the process used at any moment)
GOAL · rho >= 0.999 on the top 1M; full run < 3 h with peak RSS < 32 GB; extrapolation cuts iterations by >= 30% vs plain power iteration
Common Crawl domain-ranks file (computed with the WebGraph framework) · runtime measured locally—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

Common Crawl is a public crawl of the web that also publishes a graph of which domains link to which. Stream its latest domain-level web graph (cc-main-2026-jul-aug-sep: 133.2M nodes, 2.1B edges) into a memory-mapped CSR you build yourself (a sparse matrix stored on disk that the operating system pages into RAM as needed). Run PageRank out of core (with data larger than RAM) in NumPy on a machine with <= 32 GB RAM, in three variants: power iteration, Gauss-Seidel (uses each updated value immediately, which often converges faster) and extrapolation-accelerated PageRank (uses the last few iterates to estimate where the sequence is heading and jumps there). Then compare with Common Crawl's published domain ranks.

VICTORY CONDITIONS

  1. rho >= 0.999 on the top 1M; full run < 3 h with peak RSS < 32 GB; extrapolation cuts iterations by >= 30% vs plain power iteration (Spearman rho (a rank correlation where 1 means identical order) vs Common Crawl's published domain PageRank (top 1M); wall time; peak RSS (the most RAM the process used at any moment)).
  2. A public repository with a README that shows your result and how to reproduce it.

Terms in plain words

Memory map (memmap)
A file on disk that a program can index like an array, with the operating system loading pieces into RAM on demand.
Out of core
Processing data that is larger than RAM by streaming it from disk in pieces.
Gauss-Seidel iteration
An iterative method that uses each newly updated value immediately within the same sweep, instead of waiting for the next iteration.
Extrapolation acceleration
Combining several recent iterates to cancel the slowest-decaying error component and reach the answer in fewer iterations.
Spearman rho
The correlation between two rankings, computed on rank positions rather than raw values.

Why it rates ★★★☆☆

Impressive single-machine scale engineering, but reproducing an existing ranking is not research.