Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions

Jiarui Zhang, Ollie Liu, Tianyu Yu, Jinyi Hu, Willie NeiswangerSubmission to ICLR 2025 · Rejected

TL;DR

Current Multimodal Large Language Models (MLLMs) excel at semantic visual reasoning but struggle with basic Low-Level Visual Perception (LLVP), such as detecting whether a point lies on a line or judging line lengths. To bridge this gap, the authors introduce Geoperception, a benchmark for 2D geometric perception, systematically study MLLM design choices, and train Euclid—a 1.5B-parameter model trained entirely on synthetic geometric data that outperforms frontier models like Gemini-1.5-Pro on geometric perception tasks.

What Problem Does This Paper Address?

While modern MLLMs score high on standard visual question answering benchmarks, their success mostly relies on high-level semantic abstraction. When asked to perform foundational, low-level visual perception (LLVP)—such as reading line intersections, determining parallelism, or comparing angle sharpness—state-of-the-art models frequently fail.

This shortcoming impairs downstream visual problem solving in domains like geometry, robotics, CAD modeling, and medical imaging. The authors pinpoint a primary root cause: visual pretraining datasets inherently prioritize semantic descriptions over fine-grained geometric metadata, leaving open questions on what vision backbones, language models, and training curricula best foster LLVP.

Methods

  1. Geoperception Benchmark: A diagnostic benchmark derived from textbook geometry problems covering seven core geometric perception tasks across three categories:
  2. Logical: PointLiesOnLine, PointLiesOnCircle
  3. Numerical: AngleClassification, LineComparison
  4. Annotated: Parallel, Perpendicular, Equals
  5. Controlled Empirical Exploration: Using an MLLM architecture (frozen vision backbone + 2-layer MLP projection + autoregressive LLM), the authors isolate design choices:
  6. LLM Scale: Testing Qwen-2.5 across 0.5B, 1.5B, and 3B parameters.
  7. Vision Architectures: Evaluating CNNs (ConvNeXt-Large, ConvNeXt-XXLarge) against Vision Transformers (ViT-L/H/g, SigLIP, DINOv2).
  8. Encoder Tuning: Comparing frozen vs. fully fine-tuned visual encoders.
  9. Data Regimes: Comparing single-difficulty training, mixed-difficulty training, and dynamic curriculum learning.
  10. The Euclid Model: Armed with empirical best practices, the authors built Euclid, combining a frozen ConvNeXt vision encoder with a Qwen-2.5-1.5B-Instruct language model, trained purely on synthetically generated 2D diagrams using a dynamic multi-stage curriculum.

Datasets

  • Geoperception (Evaluation): Built on the high-school textbook corpus Geometry-3K. After filtering diagrams with missing point labels via GPT-4o-mini validation, the benchmark retains 1,584 diagrams and thousands of evaluation questions across seven basic geometric tasks.
  • Synthetic Geometry Dataset (Training): Built on top of the AlphaGeometry engine. It programmatically renders 2D geometry figures with known coordinate ground truths, generating corresponding templated question-answer pairs at three progressive complexity levels. A total of 1.6M synthetic instances were generated.

Key Insights and Ideas

  • Scaling LLMs does not improve LLVP: Scaling language backbones from 0.5B to 3B parameters yields no consistent speedup or accuracy boost on low-level geometric perception when language templates are simple.
  • CNNs outperform Vision Transformers: ConvNeXt models consistently learned faster and achieved higher accuracy than ViTs that were 3–5× larger. The authors hypothesize that ViT patch tokenization discards fine spatial details, whereas CNN convolutions preserve continuous, low-level coordinate structures.
  • Tuning vision backbones provides negligible gains: Freezing pretrained vision encoders performs just as well as fine-tuning them, showing that standard visual encoders already preserve adequate geometric signal—the bottleneck is how the LLM interprets the projected features.
  • Curriculum learning is strictly necessary: Training directly on complex ("hard") diagrams caused complete failure to converge on tasks like PointLiesOnLine. Progressively scheduling from simple to complex shapes with exponential sample smoothing enabled models to master complex shapes.

Key Results

  • Euclid Outperforms Closed-Source Frontier Models: Despite utilizing only a 1.5B LLM backbone and synthetic data:
  • Euclid-ConvNeXt-XXLarge achieved 67.89% overall accuracy on Geoperception, comfortably beating Gemini-1.5-Pro (56.98%), Claude 3.5 Sonnet (51.30%), and GPT-4o (49.68%).
  • On PointLiesOnLine, Euclid achieved 82.98%, more than triple Gemini-1.5-Pro's 24.42% and Pixtral-12B's 24.63%.
  • On numerical perception (LineComparison and AngleClassification), Euclid scored over 90%, outperforming all proprietary baselines.

Limitations and Future Work

  • Annotation Sensitivity: Euclid’s performance drops on heavily human-annotated diagrams (scoring only 31.94% on the Equals task). For instance, variable annotations (e.g., an "x" marking an angle) can mislead the model into misidentifying points.
  • Linguistic Diversity: Text generation relies on rigid prompt templates, potentially limiting generalization to open-ended conversational queries.
  • Automated Curriculum: Difficulty stages were manually designed based on geometric primitives; developing automated curriculum progression (e.g., via training loss or hard-negative mining) remains an open direction.
  • Domain Scope: Experiments focused exclusively on 2D geometry; validating whether these LLVP insights transfer to natural imagery, robotics, and CAD remains future work.

Summary updated

Abstract

Multimodal large language models (MLLMs) have made rapid progress in recent years, yet continue to struggle with low-level visual perception—particularly the ability to accurately describe the geometric details of an image. This capability is crucial for applications in areas such as robotics, medical image analysis, and manufacturing. To address this challenge, we first introduce Geoperception, a benchmark designed to evaluate an MLLM’s ability to accurately transcribe 2D geometric information from an image. Using this benchmark, we demonstrate the limitations of leading MLLMs, and then conduct a comprehensive empirical study to explore strategies for improving their performance on geometric tasks. Our findings highlight the benefits of certain model architectures, training techniques, and data strategies, including the use of high-fidelity synthetic data and multi-stage training with a data curriculum. Notably, we find that a data curriculum enables models to learn challenging geometry understanding tasks which they fail to learn from scratch. Leveraging these insights, we develop Euclid, a family of models specifically optimized for strong low-level geometric perception. Although purely trained on synthetic multimodal data, Euclid shows strong generalization ability to novel geometry shapes. For instance, Euclid outperforms the best closed-source model, Gemini-1.5-Pro, by up to 54.52% on benchmark tasks.
BibTeX
@misc{zhang2024euclid,
  author = {Jiarui Zhang and Ollie Liu and Tianyu Yu and Jinyi Hu and Willie Neiswanger},
  title = {{Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions}},
  year = {2024},
  eprint = {2412.08737},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.08737},
}
Figure