R-CoT: Reverse Chain-of-Thought Problem Generation for Geometric Reasoning in Large Multimodal Models

linger Deng, Yuliang Liu, Bohan Li, Dongliang Luo, Liang Wu, Chengquan Zhang, Pengyuan Lyu, Ziyang Zhang, Gang Zhang, Errui Ding, Yingying Zhu, Xiang BaiSubmission to ICLR 2025 · Withdrawn

TL;DR

Solving visual geometry problems remains a major blind spot for Large Multimodal Models (LMMs) due to the lack of diverse, high-fidelity training data. To bridge this gap, the authors introduce R-CoT (Reverse Chain-of-Thought), a two-stage pipeline that renders realistic geometric figures with detailed structural descriptions, computes valid multi-step deductions first, and then synthesizes questions backward from the solution. Fine-tuning models on the resulting GeoMM dataset sets new state-of-the-art benchmarks at 2B, 7B, and 8B scales, with R-CoT-8B outperforming GPT-4o by an average of 13% across MathVista and GeoQA.

What Problem Does This Paper Address?

While modern vision-language models excel at general visual question answering (VQA), they routinely stumble on elementary geometry. Existing geometric datasets suffer from critical trade-offs:

  • Rewording existing data: Using LLMs to paraphrase human-annotated datasets (e.g., G-LLaVA) expands text volume but leaves image and concept diversity stagnant.
  • Template-based data engines: Programmatic engines (e.g., GeomVerse) produce geometrically accurate figures, but the images lack the visual complexity/fidelity of real exam problems, and the rigid question templates lack linguistic variety.
  • Direct LMM generation: Prompting advanced vision models to inspect images and draft Q&A pairs introduces severe visual hallucinations and arithmetic mistakes.

The authors address the root cause: synthetic data generation needs the geometric precision of a programmatic engine paired with the linguistic diversity of LLMs—without letting the model hallucinate unanswerable questions.

Methods

The authors propose R-CoT, a two-stage data generation framework:

[Substrate Pool + Line Operations] 
           │
           ▼
     GeoChain Engine ───► High-Fidelity Geometry Image
           │
           ▼ Detailed Structural Descriptions (Ground Truth)
┌────────────────────────────────────────────────────────┐
│ Reverse A&Q (Text-only LLM reasoning via ERNIE Bot 4.0)│
│  1. Description Patch Reasoning (single-step facts)   │
│  2. Chain-of-Thought Fusion (multi-step deductions)   │
│  3. Question Generation (work backwards to question)  │
└────────────────────────────────────────────────────────┘
           │
           ▼
    Solvable, Multi-Step Geometric Q&A Pair
  1. GeoChain (Image & Description Engine): Samples from a pool of 20 geometric substrates (circles, regular polygons, triangles) and combines them iteratively. Unlike past engines, it introduces realistic auxiliary lines (midlines, chords, radii, secants) and angle annotations, while outputting an exact ground-truth textual description capturing spatial relationships (e.g., collinearity, perpendicularity, tangency).
  2. Reverse A&Q (Reverse Question Formulation): Rather than asking an LLM to pose a question and solve it, Reverse A&Q feeds the text-only structural description into an LLM through three structured sub-steps:
  3. Description Patch Reasoning: Breaks the description into small fragments and generates single-step deductions using in-context learning.
  4. Chain-of-Thought Fusion: Progressively chains single-step reasoning outputs together into coherent multi-step deductions.
  5. Question Generation: Extracts the final deduction target and works in reverse to draft a clear, solvable question corresponding precisely to the derived multi-step solution.

Because answers and deductive steps are established before formulating the question, the pipeline circumvents hallucinations and insolvable problems.

Datasets

  • GeoMM (Introduced): A synthetic dataset containing 87k high-fidelity geometric image–Q&A pairs spanning 20 shape types, 4 complexity levels, and 4 question targets (lengths, angles, areas, and element relationships).
  • Training Baselines: Models were fine-tuned using Geo170K combined with GeoMM (or competing synthetic sets: GeomVerse and the synthetic portion of MAVIS).
  • Evaluation Benchmarks: Evaluated on the testmini split of MathVista and the test set of GeoQA using standard Top-1 accuracy.

Key Insights and Ideas

  • Answer-First Inversion Prevents Hallucination: LLMs often fail to estimate problem difficulty when generating geometric questions from scratch, yielding under-constrained or unanswerable prompts. Generating the deductive chain first guarantees that every synthesized question has a strictly verified, step-by-step solution.
  • Relational Awareness Matters More than Raw Polygons: Real geometry problems depend heavily on relationships formed by auxiliary lines (e.g., midpoints, tangents). Explicitly modeling element relationships in GeoChain directly translates to better fine-grained visual perception in downstream models, preventing errors like misidentifying alternate interior angles.
  • Training Stability and Data Quality: Training on existing synthetic datasets (GeomVerse, MAVIS) degraded MathVista accuracy beyond 50k samples due to limited diversity. In contrast, GeoMM scaled smoothly up to 87k with significantly lower population variance across multiple training runs.
  • Invariance to Shape Distribution: Varying the proportion of circles, triangles, or quadrilaterals by up to ~15% shifted downstream test accuracy by less than 0.5%, indicating that performance gains stem from learning underlying deductive geometry reasoning rather than memorizing shape statistics.

Key Results

Fine-tuning multiple open-source LMM architectures with GeoMM consistently boosted performance across the board without adding extra model parameters:

Model MathVista (Top-1 Acc %) GeoQA (Top-1 Acc %)
Closed-Source Reference
GPT-4o 60.6 61.4
Open-Source Mathematical SOTA (Previous)
Math-LLaVA-13B 56.5 47.8
MAVIS-7B — 66.7
R-CoT Models (Ours)
R-CoT-2B (Mini-Monkey-2B) 57.7 (+2.4 over base) 62.6 (+0.8 over base)
R-CoT-7B (InternLM-XC2-7B) 62.0 (+3.8 over base) 67.8 (+4.0 over base)
R-CoT-8B (InternVL-2.0-8B) 73.1 (+2.0 over base) 75.9 (+1.7 over base)
  • R-CoT-8B set new open-source state-of-the-art marks: outperforming the previous open-source mathematical SOTA by +16.6% on MathVista and +9.2% on GeoQA.
  • It outperformed GPT-4o by +12.5% on MathVista and +14.5% on GeoQA.

Limitations and Future Work

  • Linguistic Redundancy / Non-Unique Mappings: Because the pipeline relies on an LLM to generate reasoning chains from structural text, identical descriptions can occasionally generate ambiguous or non-unique Q&A pairs.
  • Scope Limited to Planar Geometry: GeoChain currently focuses on 2D Euclidean geometry problems and does not yet cover solid (3D) geometry or coordinate-function graphs.
  • Future Directions: The authors plan to adapt the Reverse CoT formulation to other diagram-heavy STEM tasks (e.g., physics diagrams, function plots) while further investigating methods to curb visual hallucination in multimodal backbones.

Summary updated

Abstract

Existing Large Multimodal Models (LMMs) struggle with mathematical geometric reasoning due to a lack of high-quality image-text paired data. Current geometric data generation approaches, which apply preset templates to generate geometric data or use Large Language Models (LLMs) to rephrase questions and answers (Q&A), unavoidably limit data accuracy and diversity. To synthesize higher-quality data, we propose a two-stage Reverse Chain-of-Thought (R-CoT) geometry problem generation pipeline. First, we introduce GeoChain to produce high-fidelity geometric images and corresponding descriptions highlighting relations among geometric elements. We then design a Reverse A&Q method that reasons step-by-step based on the descriptions and generates questions in reverse from the reasoning results. Experiments demonstrate that the proposed method brings significant and consistent improvements on multiple LMM baselines, achieving new performance records in the 2B, 7B, and 8B settings. Notably, R-CoT-8B significantly outperforms previous state-of-the-art open-source mathematical models by 16.6% on MathVista and 9.2% on GeoQA, while also surpassing the closed-source model GPT-4o by an average of 13% across both datasets.
BibTeX
@misc{deng2024r,
  author = {linger Deng and Yuliang Liu and Bohan Li and Dongliang Luo and Liang Wu and Chengquan Zhang and Pengyuan Lyu and Ziyang Zhang and Gang Zhang and Errui Ding and Yingying Zhu and Xiang Bai},
  title = {{R-CoT: Reverse Chain-of-Thought Problem Generation for Geometric Reasoning in Large Multimodal Models}},
  year = {2024},
  eprint = {2410.17885},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.17885},
}
Figure