Abstract
Geometric reasoning remains a core challenge for Multimodal Large Language Models (MLLMs). Even the most advanced closed-source systems, such as GPT-O3 and Gemini-2.5-Pro, still struggle to solve geometry problems reliably, despite exhibiting strong textual reasoning abilities on tasks like the International Mathematical Olympiad (IMO). This gap suggests that the bottleneck lies in understanding geometric diagrams rather than reasoning itself. Since geometric figures can often be faithfully described in concise textual form, converting visual content into captions offers a promising direction. Motivated by this insight, we introduce CapGeo, a caption-assisted reasoning framework that bridges visual and textual modalities. Experiments show substantial improvements when models are equipped with captions: Qwen2.5-VL-72B improves from 8.6% (vision-only) to 59.0%, while Claude-Opus-4 rises from 44.8% to 73.0%. To systematically evaluate and identify high-quality geometric captioning models, we further propose CapGeo-Bench, a dataset of 4,641 curated figure-caption pairs. Crucially, CapGeo-Bench incorporates a keypoint-based evaluation metric that correlates strongly with downstream CapGeo performance, enabling reliable assessment of geometric captioning ability. Together, our framework and benchmark highlight a new pathway toward advancing geometric reasoning in MLLMs.
1. Introduction
Multimodal Large Language Models (MLLMs) have achieved remarkable success in text-based mathematical reasoning, yet they consistently struggle with middle-school geometry problems. While models like GPT-O3 demonstrate high-level logical capabilities, their performance on geometric tasks is often hampered by visual perception errors. Geometric diagrams are abstract and symbolic; a minor misinterpretation of a line or a misread numerical label can invalidate an entire proof. This paper identifies this "perception bottleneck" as the primary obstacle and proposes CapGeo, a framework that leverages structured textual captions to ground visual reasoning.
2. The CapGeo Framework
CapGeo is a two-stage reasoning framework designed to bypass the limitations of direct visual parsing. In the first stage, a captioning model generates a rigorous, formal description of the diagram, detailing points, lines, intersections, and numerical values (e.g., "In circle O, AB is a diameter... angle BOD equals 140°"). In the second stage, this structured caption is provided alongside the original image and question to a reasoning model.
By converting noisy pixels into high-density textual tokens, CapGeo aligns geometric information with the pre-trained reasoning strengths of LLMs. This process forces a "grounding" of the reasoning, making the final inference significantly more robust against visual hallucinations and misinterpretations.
3. CapGeo-Bench: Dataset and Evaluation
To support the development of high-quality geometric captioning, we introduce CapGeo-Bench, a comprehensive dataset consisting of 4,641 figure-caption pairs. The data is curated from K-12 textbooks and covers plane, solid, and analytic geometry across four difficulty levels. Each entry was meticulously annotated by STEM students from top universities and verified by mathematics experts.
To evaluate caption quality, we developed a Keypoint-by-Keypoint Evaluation Framework. This metric assesses captions across three critical dimensions:
- Elements: Identification of all geometric primitives (points, lines, etc.).
- Spatial Relations: Accuracy of geometric relationships (e.g., perpendicularity, parallelism).
- Numericals: Precision in extracting lengths and angles.
4. Results and Key Insights
Experimental results across benchmarks such as MathVerse, MathVista, and GeoQA demonstrate that caption assistance provides transformative performance gains. Qwen2.5-VL-72B’s accuracy on "Vision-Only" tasks improved from 8.6% to 59.0% with captioning support. Similarly, Claude-Opus-4 saw its performance rise from 44.8% to 73.0%.
Key insights from our analysis include:
- Perception vs. Reasoning: The massive performance jump confirms that models possess the necessary mathematical logic but fail due to unreliable visual extraction.
- Numerical Weakness: Even state-of-the-art models struggle with the "Numerical" dimension of captioning, particularly at higher difficulty levels.
- Metric Correlation: We observed a strong positive correlation between the "Keypoint Score" and final problem-solving accuracy, validating our evaluation methodology.
5. Conclusion
CapGeo highlights a promising pathway for advancing geometric reasoning by bridging the gap between visual perception and textual logic. While the current two-stage approach introduces inference overhead, it provides a robust baseline for future research. Subsequent efforts should focus on improving numerical extraction and exploring end-to-end training methods that internalize structured captioning logic within MLLM architectures.
Summary updated