Introduction
The rapid advancement of Large Language Models (LLMs) has necessitated the creation of increasingly difficult benchmarks to evaluate their reasoning capabilities. While benchmarks like GSM8k and MATH have become industry standards for English-language reasoning, there remains a significant gap in evaluating models on expert-level STEM tasks in other languages. The DOoM (Difficult Olympiads of Math) benchmark addresses this by providing a rigorous evaluation suite for mathematics and physics problems in Russian, sourced from prestigious academic competitions and university entrance exams.
Methodology and Verifiability
A primary challenge in developing a high-level reasoning benchmark is ensuring the accuracy of automated grading. To handle the complex symbolic and numerical outputs typical of Olympiad-level problems, the authors implemented a multi-stage verification pipeline:
- Format Normalization: Initial comparison of model outputs as fractions.
- Numerical Validation: Checking for equivalence as natural numbers.
- Symbolic Parsing: Utilizing LaTeX and Python expression parsing to determine if a model's symbolic answer is mathematically equivalent to the ground truth.
This robust pipeline allows for the evaluation of a diverse range of models, from proprietary frontier systems like Gemini and GPT-o3 to open-source weights like Qwen and Gemma, as well as Russian-native models like GigaChat.
Dataset Composition
The DOoM benchmark is divided into two core domains:
- RussianMath (62.1%): This subset spans topics from combinatorics and arithmetic progressions to advanced coordinate geometry.
- RussianPhysics (37.9%): This subset focuses on mechanics, kinematics, and thermodynamics.
The problems are sourced from high-stakes environments, including the All-Russian Olympiad and entrance exams for elite institutions such as Moscow State University (MSU), the Moscow Institute of Physics and Technology (MIPT), and the Higher School of Economics (HSE).
Key Insights: The "Physics Gap" and the Value of "Thinking"
Evaluation results revealed several critical trends in LLM reasoning:
The Physics Gap
Across all tested models, performance in physics was consistently lower than in mathematics. This suggests that while models can master the formal rules of mathematical manipulation, they struggle with the "qualitative mental models" required to translate physical scenarios into solvable equations. This gap may also be a result of data bias, as high-quality, step-by-step physical reasoning is less prevalent in common training corpora compared to structured mathematical proofs.
The "Thinking" Correlation
One of the most significant findings is the strong positive correlation ($r = 0.707$) between the number of tokens a model generates and its final score. This "thinking longer" effect confirms that models employing extensive Chain-of-Thought (CoT) reasoning are significantly more successful at solving complex problems.
Comparative Performance
The benchmark results establish a clear hierarchy of current LLM capabilities in the Russian STEM context. Gemini 2.5 Pro emerged as the top performer, followed closely by OpenAI's reasoning-optimized models. Notably, while Russian-native models like GigaChat 2 Max lead among localized systems, they still face a substantial performance gap compared to global frontier models.
Conclusion and Future Directions
The DOoM benchmark provides a vital tool for measuring the reasoning depth of LLMs in a non-English technical context. Future work will focus on expanding the dataset's complexity to prevent model saturation and refining the evaluation of multi-step proofs that cannot be easily reduced to a single answer. By open-sourcing the benchmark, the authors aim to foster community collaboration in pushing the boundaries of automated STEM reasoning.
Summary updated