While Large Vision-Language Models (LVLMs) excel at describing natural scenes, they frequently falter when interpreting data visualizations. As noted by Chen et al. and Ji et al., these models often hallucinate numerical values or misinterpret trends because they attempt to perform complex logical reasoning directly on raw pixels. To bridge this gap, researchers have converged on a strategy known as de-rendering: the process of reconstructing structured, symbolic data from visual representations.
The Perception-Reasoning Gap
The fundamental challenge in chart understanding is that it requires two distinct capabilities: precise visual perception (identifying the exact height of a bar or the slope of a line) and logical reasoning (performing arithmetic or comparing categories). General-purpose models often conflate these steps. Liu et al. argue that by decomposing these tasks, we can leverage the superior reasoning capabilities of text-based Large Language Models (LLMs).
By converting a chart into a linearized table (such as a Markdown or CSV format), the visual data is "de-rendered" into a language the LLM already understands. This transformation allows the model to treat a visual reasoning problem as a table-based reasoning problem, a domain where LLMs are already highly proficient.
Structured Data as a Factual Safety Net
De-rendering serves as more than just a translation layer; it acts as a "factual safety net." Chen et al. demonstrate that providing a language model with an explicit tabular representation of a chart significantly reduces numerical hallucinations. Instead of guessing a value based on visual features, the model references a "hard copy" of the underlying numbers.
This modular approach is particularly effective in specialized domains. For example, Chang et al. introduced the V-MODEQA framework for choropleth maps, which first extracts structured data from map images before attempting logical reasoning. Similarly, Khalighinejad et al. found that employing specialized models to transform scientific plots into intermediate tables reduces the noise and complexity that general-purpose models typically encounter.
Learning the Mapping from Pixels to Tables
For a model to de-render effectively, it must learn the precise relationship between visual geometry and semantic labels. Several researchers have proposed pre-training objectives specifically designed to force this alignment:
- Chart-Table Pairs: Zhou et al. developed ChartT5, which uses "fill-in-the-blank" tasks on chart headers and numerical values to teach the model how pixels map to structured data.
- Visual-to-Numeric Mapping: Fan et al. show that by emphasizing raw data extraction during training, models can learn to interpret data from visual geometry—like bar heights—even when explicit numerical labels are missing.
- Structural Alignment: Meng et al. and Masry et al. argue that reconstructing the underlying data table is a superior pre-training objective. This "bottleneck task" forces the model to establish a precise correspondence between 2D pixel coordinates and logical data hierarchies before it ever attempts to answer a question.
From Pixels to Executable Logic
The most advanced iterations of this idea move beyond simple tables to executable code. Ma et al. describe "vision-centric OCR," where complex charts and diagrams are converted into executable formats that maintain the original's functional hierarchy.
Similarly, Sahu et al. and Li et al. propose using a "Program of Thought" approach. In these frameworks, the visual data is extracted into a queryable format (like a Python dataframe), and the reasoning is performed by executing generated code. This ensures that the logical operations—such as calculating a percentage change or finding a maximum—are handled by a deterministic execution engine rather than a probabilistic language model.
By treating visual data as a structured intermediate representation, these methods transform the "black box" of visual reasoning into a transparent, two-stage process: accurate perception followed by rigorous logical analysis.
Key Foundational Work
Converting Basic D3 Charts into Reusable Style Templates (2018) — contributed 2 concept elements to this direction.
Answering Questions about Data Visualizations using Efficient Bimodal Fusion (2020) — contributed 1 concept elements to this direction.