When humans look at a chart, a table, or a scientific diagram, we do not just see a collection of pixels or a holistic scene. We instantly decode a complex, non-linear data structure. We map a bar's height to an axis value, correlate a line color with a legend entry, and trace rows and columns to find a specific data point.
Modern Vision-Language Models (VLMs) excel at holistic scene understanding—identifying a cat in a park or describing a landscape. However, as a growing body of research shows, they fundamentally struggle when forced to interpret structured visual data. Understanding why general-purpose VLMs fail at these tasks, and how researchers are re-engineering them to bridge this gap, reveals a core tension between semantic perception and precise structural reasoning.
The Perceptual Mismatch: Scene vs. Structure
The root of the problem lies in how standard visual encoders are trained. As noted by Zhengzhuo Xu et al. in their work on ChartMoE, general-purpose visual encoders are optimized for holistic scene understanding. They compress images into spatial patch tokens that capture broad semantic features.
This compression is disastrous for structured data like charts and tables, which function as visual encodings of precise, abstract data. When a VLM treats a chart as a generic image, it often fails to ground its output in the actual underlying data, leading to a high risk of misinterpreting critical visual elements like legends and axes (Daoyuan Chen et al.).
Natural Image (Holistic) Structured Visual Data (Granular)
+--------------------------+ +--------------------------+
| [Dog] [Ball] | | Y-Axis |* (Data Point) |
| (Spatial relationships | | Scale | |
| are fluid and fuzzy) | | +------- X-Axis |
+--------------------------+ +--------------------------+
Result: High VLM Success Result: High VLM Failure (Requires
exact pixel-to-value mapping)
Because the visual features are blurred across patches, the model loses the fine-grained spatial awareness required for quantitative tasks. This manifests across several domains:
- Engineering and Materials Science: Models fail to achieve the numerical accuracy necessary for detailed design, often hallucinating data points or omitting critical details from technical charts (Cyril Picard et al.; Defne Circi et al.).
- Meteorology: Complex heatmaps with irregular shapes and varied color gradients confuse standard architectures, hindering automated weather analysis (Jian Chen et al.).
- Technical Captioning: Open-source generalist captioning models lack the fine-grained perception needed for mathematical diagrams, producing descriptions that are too vague for academic use (Zhenxin Lei et al.).
The Cascading Error Problem
When a VLM is tasked with converting a visual chart into a structured text format like JSON, CSV, or a data table, it faces a strict constraint: perfection is required.
As Fuxiao Liu et al. observe in the development of the MMC benchmark, converting visual charts into structured formats requires flawless optical character recognition (OCR) combined with exact spatial mapping. In a natural language description, a single wrong word might not destroy the overall meaning. But in a structured data table or a Global Trade Item Number (GTIN), a single digit error or a misaligned column invalidates the entire output (Bianca Lamm et al.).
This difficulty is compounded when models must extract data from complex scientific charts into structured tables. In evaluations of state-of-the-art VLMs on the Chart2CSV benchmark, Chuxuan Hu et al. found that models misinterpret nearly half the data. The models struggle not just with extracting individual data points, but with understanding the broader data organization and complex visual relationships, such as mapping confidence intervals back to their corresponding variables.
The Table Parsing Bottleneck
If charts are difficult, tables presented as images are a primary bottleneck for multimodal architectures. Standard text processing is linear (left-to-right, top-to-bottom). Tables, however, require non-linear, multi-directional spatial reasoning to decode the structural relationships between rows and columns.
Linear Text Processing:
[Token 1] -> [Token 2] -> [Token 3] -> [Token 4]
Tabular Spatial Processing (Non-Linear):
Header 1 <---> Header 2
^ ^
v v
Row 1, Col 1 <-> Row 1, Col 2 (Requires explicit row/col alignment)
Research by Rocktim Jyoti Das et al. on the EXAMS-V benchmark demonstrates that multimodal models suffer a significant performance drop when processing structured data like tables and graphs compared to standard pictorial figures.
The underlying reason for this failure is explored by Congyun Jin et al. in the medical domain (RJUA-MedDQA). They show that direct visual processing by end-to-end multimodal models often fails to capture the structural relationships in dense, multi-column tables. Without an explicit step to restore or map the document's layout, the model lacks the spatial awareness required to link a row entry to its correct column header. This is why parsing high-resolution, image-based tables remains a massive hurdle for automated scientific reasoning (Rama Chellappa et al.; Jaewoo Lee et al.).
Re-Architecting the Vision-Language Interface
To overcome these limitations, researchers are moving away from treating chart interpretation as a standard image-to-text problem. Instead, they are developing architectures and training paradigms explicitly designed to preserve and decode structure.
Diversely Aligned Connectors
Rather than forcing a general visual encoder to change, Zhengzhuo Xu et al. proposed ChartMoE, which introduces a Mixture of Experts (MoE) architecture into the connector—the bridge that translates visual tokens into the language model's text space. By initializing different experts through specialized alignment tasks (such as translating charts directly to tables, JSON, or Python code), the model learns to extract granular data points and trends without losing its ability to understand general scenes.
Bypassing Intermediate Representations
While some approaches rely on a pipeline that first converts an image to a table or code, this can introduce intermediate errors. To counter this, Viviana Ventura et al. introduced QwenChart-7B, an instruction-tuned model that leverages dynamic Chain-of-Thought (CoT) prompting. By training the model to reason step-by-step about the visual relationships directly on the image, it achieves high accuracy on complex scientific figures without relying on an intermediate code-generation step.
Bridging the Data-Transformation Gap
There is often a disconnect between a model's ability to generate code and its ability to understand the underlying data manipulation. As Nan Chen et al. discovered using the VisEval benchmark, models frequently fail to implement requested data transformations (like specific sorting orders) even when the overall chart structure looks correct.
To fix this, modern training paradigms increasingly incorporate structured language pretraining. For instance, in cross-modal retrieval, the authors of VLPCook found that transforming flat image-text pairs into explicitly structured recipe formats allowed the model to better map complex, multi-component visual data to its textual counterpart (Mustafa Shukor et al.).
By treating structured visual data interpretation as a distinct problem requiring specialized spatial alignment, explicit layout awareness, and rigorous structural training, the field is moving closer to models that can truly "see" and reason about the data that underpins our visual world.
Key Foundational Work
EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models (2024) — contributed 3 concept elements to this direction.
SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers (2024) — contributed 3 concept elements to this direction.
Instruction-tuned QwenChart for Chart Question Answering (2025) — contributed 2 concept elements to this direction.