As Large Language Models (LLMs) achieve near-perfect scores on standard benchmarks like GSM8K and MATH, the research community faces a "saturation" problem. While the International Mathematical Olympiad (IMO) offers a higher ceiling, evaluating complex proofs often introduces subjective "noise" from LLM judges. RIMO (Remade International Mathematical Olympiad) addresses this by converting elite competition problems into deterministic, easy-to-grade formats, exposing a significant reasoning gap in even the most advanced frontier models.
The RIMO Framework: N and P Tracks
RIMO introduces a two-track structure designed to eliminate evaluation ambiguity while maintaining the "peak difficulty" of original Olympiad problems. To ensure accuracy, the ground truth was cross-referenced across official IMO shortlists, community wikis like AoPS, and expert expositions.
- RIMO-N (Numeric Track): This track consists of 335 IMO problems (1959–2023) meticulously rewritten to yield a single, unique integer answer. This allows for "O(1) string-match grading," removing the need for inconsistent LLM judges.
- RIMO-P (Proof Track): For 456 proof-based problems, the authors decomposed each into a sequence of guided sub-problems or lemmas. This provides a granular view of a model's deductive chain, rather than a binary pass/fail on a long-form proof.
Key Insights: The Reasoning Gap
The study reveals several critical characteristics of current mathematical LLMs:
- The Proof Gap: High performance on numeric answer-finding (RIMO-N) does not guarantee success in rigorous proof-writing (RIMO-P), suggesting a lack of deep deductive consistency.
- Binary Bias: Models show an 8-30% performance boost on problems with binary (0 or 1) answers, suggesting they often "luck" into correct answers when the search space is small. RIMO-N's larger integer space forces true calculation.
- Optimization Over Scale: Explicit reasoning optimization, as seen in models like DeepSeek-R1, provides significantly higher gains (up to 19.4 percentage points) than simply increasing model parameters.
Benchmarking Results
The most striking finding is the performance collapse when moving from standard high school tests to RIMO. While frontier models like GPT-4o excel at GSM8K (~95%) and MATH (~90%), their performance on RIMO-N is substantially lower.
- Top Performers: DeepSeek-R1-671B led the pack with 62.96% on RIMO-N, followed by Gemini 2.5 Flash at 58.81%.
- The GPT-4o Gap: Despite its general dominance, GPT-4o achieved only 33.43% on RIMO-N, highlighting the difficulty of true Olympiad-level reasoning.
- Small Model Efficiency: Specialized smaller models, such as Qwen3-8B (36.72%), outperformed much larger generalist models, proving the value of targeted domain pre-training.
Future Directions
While RIMO-N provides a noise-free metric, RIMO-P still utilizes LLM judges for sub-step evaluation. Future work aims to translate RIMO-P into formal verification languages like LEAN, enabling machine-verifiable proofs—the "gold standard" of mathematical correctness. Additionally, the team plans to expand the leaderboard and perform deeper error analysis to pinpoint exactly where logical chains fail in current models.
Summary updated