Theorem Reranker
Describe a theorem in English, LaTeX or a mix of both, and get back the matching declaration from Mathlib, Lean's library of formalised mathematics.
Overview
Mathlib has over a hundred thousand theorems, and finding the one you need usually means already knowing its name. This project searches all 136,608 of them from a plain description. A fine-tuned embedding model narrows the corpus to 100 candidates, then a cross-encoder reads the top 20 closely and puts them in order.
On a 151-query benchmark I wrote, the right theorem comes first 79.5% of the time and appears in the top 10 for 94.7% of queries. The best off-the-shelf baseline gets 56.3% first.
Results
- Embedding model + reranker 79.5%right theorem ranked first
- Fine-tuned embedding model alone 59.6%ranked first
- Best baseline (BM25 + embeddings, fused) 56.3%ranked first
How it works
- Each theorem is indexed with its Lean statement, docstring and an informal English statement from the FrenzyMath dataset.
- A bge-base embedding model, fine-tuned with contrastive learning on 74,833 pairs, embeds the query and scans every theorem vector for the 100 closest.
- A cross-encoder reranker, fine-tuned to notice one-symbol differences, reads the query next to each of the top 20 and re-sorts them.
- A Gradio app serves both stages, with a switch to compare against the first stage alone.
Decisions worth explaining
A benchmark built to catch near misses
A third of the 151 queries are traps: each comes with near-miss theorems such as the converse, the strict version of an inequality, or the same statement without a hypothesis. The evaluation reports how often a near miss is ranked above the real answer, which a plain accuracy number would hide.
Checking the data before trusting it
The informal statements are written by a language model. A spot check of 50 found 12 subtly wrong, mostly missing a hypothesis, so the benchmark queries are written from the Lean statements, which are exact. Every theorem in the benchmark is held out of training.
Why there are two stages
I first tried teaching the embedding model to tell a theorem from its one-symbol edit. It learned to (97% accuracy on held-out pairs), but accuracy on the real benchmark dropped sharply: pushing near-identical theorems apart distorted the space every query shares. So that job moved to a reranker that reads the query and a candidate together, and only has to do it for 20 candidates.
Measuring the cost
On two CPU threads, the first stage answers in about 60 ms and the reranker brings the median to about 1.1 seconds. Looking deeper than the top 20 didn't help: more candidates just meant more near twins to get wrong.