Chain-of-region: Visual Language Models Need Details for Diagram Analysis

Xue Li, Yiyou Sun, Wei Cheng, Yinglun Zhu, Haifeng ChenICLR 2025 · OpenReview

The "Near-Sighted" Challenge of VLMs

While state-of-the-art Visual Language Models (VLMs) like GPT-4V and Gemini demonstrate impressive general scene understanding, they often struggle with the precise nature of scientific diagrams. When tasked with reading exact values from bar charts, identifying thin lines in circuit diagrams, or parsing complex spatial relationships, these models frequently hallucinate or miss critical fine-grained details.

The authors of this paper argue that this is a "perceptual" failure rather than a reasoning one. Current models typically process images in a single pass, which glosses over the meticulous details required for rigorous scientific analysis. To address this, the authors introduce the Chain-of-Region (CoR) framework, a pipeline designed to systematically decompose diagrams into discernible visual elements and feed structured metadata back into the model.

Chain-of-Region Framework in Action

The Chain-of-Region (CoR) Methodology

CoR operates as a "plug-and-play" pipeline utilizing a "divide-and-conquer" strategy. It avoids computationally expensive deep-learning-based segmentation, opting instead for the precision of traditional computer vision techniques.

CoR Architecture and Pipeline

The framework consists of three primary stages:

  1. Region Identification:
  2. Initialization: The system uses traditional OpenCV techniques (such as Otsu’s thresholding and connected components) to identify initial shapes.
  3. Splitting: If a region is overly complex, a VLM identifies potential structured shapes (e.g., rectangles, ellipses, lines), which are extracted using specialized "white-box" detectors. For irregular shapes, a "liquid filling" analogy using distance transforms is applied to break weak connections.
  4. Merging: To prevent the VLM from being overwhelmed by hundreds of tiny fragments, CoR merges related regions (like text captions and their associated bounding boxes) using hierarchical clustering and heuristic rules.
  5. Information Assembly: For each identified region, the system extracts Structure Info (coordinates, area, angles) and Semantic Info (using the VLM to generate a targeted description of that specific, highlighted region).
  6. Aggregation and Query: All compiled metadata is structured into a text prompt and fed back to the VLM alongside the original image for the final, comprehensive analysis.

Datasets and Evaluation

The primary evaluation leveraged the MMMU (Massive Multi-discipline Multimodal) dataset, specifically targeting 5,210 images within the Science and Tech & Engineering sub-categories. Because public datasets for scientific diagram segmentation are scarce, the authors also curated a custom ground-truth dataset of over 100 samples derived from MMMU. This enabled them to measure Mean Intersection over Union (mIoU) for both individual elements and grouped semantic regions.

Key Insights and Results

The CoR framework demonstrated dominant performance across multiple VLM backbones, yielding several critical insights:

  • Massive Accuracy Boost: When paired with GPT-4o, the CoR framework achieved an average accuracy of 52.9% on the MMMU scientific subset, a massive improvement over the 37.7% achieved by the raw VLM. It also consistently outperformed other prompting strategies like "Set-of-Mark" (SoM) and Chain-of-Thought (CoT).
  • Traditional CV Outperforms Deep Learning for Diagrams: "Old-school" computer vision (OpenCV) proved better suited for scientific diagrams than modern deep-learning segmentors like Meta's SAM2. Diagrams feature homogeneous colors and structured patterns that rule-based algorithms can process with millisecond efficiency. In segmentation tasks, CoR achieved 63.0% mIoU compared to SAM2's 42.2% in the "individual" setting.
  • The Granularity "Goldilocks" Zone: Sensitivity analysis revealed that the number of regions sent to the VLM is crucial. Too few regions cause the model to miss details, while too many overwhelm it with minutiae, causing it to lose sight of the overall structure.
  • White-Box Interpretability: Because CoR’s region separation is rule-based rather than a neural network "black box," the system is highly interpretable and easier to debug.

Limitations and Future Work

While highly effective, the authors note a few limitations. The system currently relies on a predefined library of structured detectors; if a diagram contains an unsupported shape, it must fall back on less precise unstructured splitting. Additionally, extracting hidden parameters of complex visual elements often requires difficult reverse-engineering. Future iterations of CoR may explore dynamic clustering to automatically adjust the number of regions based on diagram complexity, rather than relying on a fixed budget.

Summary updated

Abstract

Visual Language Models (VLMs) like GPT-4V have broadened the scope of LLM applications, yet they face significant challenges in accurately processing visual details, particularly in scientific diagrams. This paper explores the necessity of meticulous visual detail collection and region decomposition for enhancing the performance of VLMs in scientific diagram analysis. We propose a novel approach that combines traditional computer vision techniques with VLMs to systematically decompose diagrams into discernible visual elements and aggregate essential metadata. Our method employs techniques in OpenCV library to identify and label regions, followed by a refinement process using shape detection and region merging algorithms, which are particularly suited to the structured nature of scientific diagrams. This strategy not only improves the granularity and accuracy of visual information processing but also extends the capabilities of VLMs beyond their current limitations. We validate our approach through a series of experiments that demonstrate enhanced performance in diagram analysis tasks, setting a new standard for integrating visual and language processing in a multimodal context.
BibTeX
@inproceedings{li2025chain,
  author = {Xue Li and Yiyou Sun and Wei Cheng and Yinglun Zhu and Haifeng Chen},
  title = {{Chain-of-region: Visual Language Models Need Details for Diagram Analysis}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=M6fYrICcQs},
}
Figure