about
Goedel-Prover: A Frontier Model for Open-Source Automated Theorem Proving [pdf] (arxiv.org)
6 points by bikenaga on Feb 12, 2025 | hide | past | pdf | 2 comments on HN

In plain words: Models translate math problems into formal statements, then each new prover solves more of them, adding fresh proofs to train the next. Trained only on this data, with no reinforcement learning, it solved 57.6% of miniF2F problems, beating the prior best by 7.6 points.

Abstract · Goedel-Prover: A Frontier Model for Open-Source Automated Theorem Proving

We introduce Goedel-Prover, an open-source language model that achieves state-of-the-art (as of April 5 2025) performance in automated formal proof generation for mathematical problems. A key challenge in this field is the scarcity of formalized mathematical statements and proofs, which we address through the following approaches. First, we train LLMs to convert natural language math problems from the Numina dataset to equivalent formal statements in Lean 4. This process creates the dataset Goedel-Pset-v1, which includes 1.64 million formal statements. Next, we develop a large dataset of formal proofs by training a series of provers. Each new prover can prove many statements that previous ones could not, and these new proofs are added to the training set for the next prover. Finally, we obtain the dataset Goedel-Pset-v1-solved, which contains proofs for over 800K statements from Goedel-Pset-v1. Supervised fine-tuning (SFT) of DeepSeek-Prover-V1.5-Base on Goedel-Pset-v1-solved (i.e., no RL) yields a Goedel-Prover-SFT that achieves a success rate of 57.6% (Pass@32) on miniF2F, surpassing the previous leader DeepSeek-Prover-V1.5-RL (trained using SFT + RL on a proprietary dataset) by 7.6%. On PutnamBench, Goedel-Prover-SFT successfully solves 7 problems (Pass@512), ranking first on the leaderboard. We provide extensive discussion of our training methodology, highlighting the key design choices that contribute to Goedel-Prover's strong performance. Further RL training (including DPO) improves Goedel-Prover-SFT's success rate to over 60% (Pass@32) on miniF2F. To aid future research, we provide extensive discussion of our training methodology and design choices. We also fully open-source our codes, models, and datasets. Additionally, we open-source formal proofs for 29.7K problems in Lean Workbook, nearly doubling the 15.7K solved by prior provers.

Yong Lin, Shange Tang, Bohan Lyu, Jiayun Wu, Hongzhou Lin, Kaiyu Yang, Jia Li, Mengzhou Xia, Danqi Chen, Sanjeev Arora, Chi Jin
arXiv:2502.07640 · cs.LG, cs.AI · submitted Feb 11, 2025 · updated Apr 19, 2025
abstract · pdf · html

add comment on HN

Has Sanjeev Arora pivoted to AI? Was he always interested in this? I thought he was more of a theoretician.

Maybe he’ll crack why DNNs work, whatever that means. To answer that question you have to formalize what it means for them to “work”. Good luck defining that in a reasonable way.

The majestic genius of Gödel was finding a way of exhibiting an unprovable truth.*

* yes, assuming consistency