To build vision-language models (VLMs) capable of reading charts, documents, and diagrams, researchers face a data bottleneck. Collecting and manually annotating millions of real-world data visualizations is incredibly labor-intensive. The standard solution has been to generate data programmatically. By using code-guided pipelines—such as the framework proposed by Yue Yang et al. in Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation—or pairing structured tables with automated plotting engines as done by Ming-Yang Zhou et al. in Enhanced Chart Understanding via Visual Language Pre-training on Plot Table Pairs, we can easily produce massive, perfectly labeled datasets.
Yet, when models trained on these pristine datasets are deployed "in the wild," they encounter a "complexity wall." As demonstrated by Muye Huang et al. in EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understanding, even state-of-the-art models suffer severe performance drops when transitioning from clean synthetic benchmarks to real-world charts crawled from the web.
This breakdown happens because synthetic data generation pipelines inadvertently strip away the messy, organic irregularities of real-world visual communication.
The Anatomy of the Domain Gap
The synthetic-to-real generalization gap is not caused by a single flaw, but rather by a combination of three distinct types of real-world complexity that synthetic generators fail to replicate: visual noise, structural complexity, and stylistic diversity.
1. Physical and Digital Noise
Synthetic images are mathematically perfect configurations of pixels. Real-world documents, however, are frequently degraded by physical and digital transmission.
When a chart is printed, scanned, faxed, or photographed, it acquires artifacts like motion blur, lighting variations, folds, and stains. As François Charih et al. observe in Audiogram Digitization Tool for Audiological Reports, low-quality scans and overlapping symbols introduce a level of noise that standard vision models trained on clean data cannot parse. This transition from low-noise synthetic environments to the real world introduces severe attribute ambiguity, a challenge highlighted by Ameet Deshpande et al. in CLEVR Parser: A Graph Parser Library for Geometric Learning on Language Grounded Image Scenes.
2. Structural and Layout Complexity
Procedural chart generators typically output isolated, single-panel plots with clean backgrounds and perfectly spaced elements. Real-world scientific and journalistic figures are rarely so accommodating.
Authentic visualizations frequently feature multi-panel layouts, composite charts, intricate legends, and overlapping data points or labels. Omar Moured et al. note in AltChart: Enhancing VLM-based Chart Summarization Through Multi-Pretext Tasks that synthetic data lacks this structural variety.
Furthermore, when a model must interpret a multi-chart report or dashboard, it has to navigate semantic misalignments and mismatched axes across different figures. As Anirudh Iyengar Kaniyar Narayana Iyengar et al. show in InterChart: Benchmarking Visual Reasoning Across Decomposed and Distributed Chart Information, these multi-step integration challenges are entirely absent from simplified synthetic training sets, causing model performance to plummet when faced with professional publications.
3. Stylistic and Human Inconsistencies
Synthetic generators rely on a limited set of uniform machine fonts and rigid templates. Human designers, by contrast, introduce immense stylistic heterogeneity. They incorporate non-standard fonts, logos, hand-drawn elements, and unique infographic layouts that break standard design conventions.
Kushal Kafle et al., in Answering Questions about Data Visualizations using Efficient Bimodal Fusion, point out that this gap between clean, programmatic visualizations and "in-the-wild" data remains a primary barrier to practical deployment. The authors of VProChart: Answering Chart Question Through Visual Perception Alignment Agent and Programmatic Solution Reasoning echo this finding, noting that rigid machine-generated data cannot bridge the gap to human-curated charts where design conventions are inconsistent or entirely missing.
This issue extends beyond charts to other text-rich visual domains. For example, in Unconstrained Text Detection in Manga: A New Dataset and Baseline, Julián Del Gobbo et al. demonstrate that models trained on synthetic datasets that simply overlay machine fonts on images fail to replicate or recognize the complex artistic integration and hand-drawn variations characteristic of authentic visual media.
Bridging the Gap
To overcome these limitations, recent research focuses on two parallel tracks: exposing the boundaries of current models through authentic, real-world benchmarks, and designing training paradigms that explicitly account for real-world irregularities.
Real-World Benchmarking
To diagnose exactly where synthetic training fails, researchers are curating datasets directly from uncurated, human-designed sources:
- RealCQA: Scientific Chart Question Answering as a Test-bed for First-Order Logic: Bhavin Jawade introduced this benchmark using complex visualizations from scientific literature, revealing that models flounder when forced out of their synthetic comfort zones by overlapping labels and varied fonts.
- SciCap: Generating Captions for Scientific Figures: Ting-Yao Hsu et al. compiled over two million real-world figures from arXiv, demonstrating that capturing nuanced human insights from complex sub-figures remains a steep challenge for standard models.
- ChartCheck: Explainable Fact-Checking over Real-World Chart Images: Mubashara Akhtar et al. developed a dataset to evaluate visual reasoning against the messy, varied design patterns found in actual news and social media graphics.
Robust Training Paradigms
Rather than abandoning synthetic data entirely, researchers are finding ways to make models resilient to the real-world domain shift.
One approach is to ground models in fundamental geometric structures. In Math Blind: Failures in Diagram Understanding Undermine Reasoning in MLLMs, Yanpeng Sun et al. demonstrated that explicitly training models to perceive low-level geometric primitives and spatial relationships yields massive improvements in high-level reasoning. This structure-aware perception helps models remain robust when confronted with the visual noise and irregularities of scanned textbooks or handwritten notes.
Another strategy involves injecting authentic human design distributions directly into the training loop. In ChartMaster: Advancing Chart-to-Code Generation with Real-World Charts and Chart Similarity Reinforcement Learning, Wentao Tan et al. showed that training on diverse, human-designed visualizations from scientific literature significantly improves a model's ability to generalize. By exposing the network to a rich distribution of layouts and text styles during training, the model learns to tolerate the organic variations and inconsistencies that define the real world.
Key Foundational Work
Scatteract: Automated Extraction of Data from Scatter Plots (2017) — contributed 1 concept elements to this direction.
Softening Discrete Relaxation (1996) — contributed 1 concept elements to this direction.