ChartCards: A Chart-Metadata Generation Framework for Multi-Task Chart Understanding

Yifan WuarXiv 2025 · DOI · arXiv

TL;DR

This paper introduces ChartCards, a novel framework for automatically generating rich metadata for charts, enabling a single chart to support multiple downstream understanding tasks. They leverage this framework to create MetaChart, a large-scale, high-quality dataset that significantly improves the performance of multimodal large language models (MLLMs) on tasks like text-to-chart retrieval and chart-to-table conversion.

What Problem Does This Paper Address?

Understanding charts is crucial for extracting information from data, but it's a complex task that often requires specialized models and large, task-specific datasets. Existing approaches struggle with the fine-grained nature of chart understanding, leading to high data collection and training costs. Furthermore, many existing datasets are limited in their task diversity, dataset size, or quality validation, hindering the development of robust, multi-task chart understanding systems. This paper aims to bridge this gap by providing a scalable framework for generating comprehensive chart metadata and a corresponding large-scale dataset to facilitate multi-task chart understanding.

Methods

The core of this work is the ChartCards framework, which automates the generation of chart metadata. This metadata includes:

  • Data tables: The underlying numerical data of the chart.
  • Visualization code: The code used to generate the chart (e.g., Matplotlib, Vega-Lite).
  • Visual elements: Information about chart components like titles, axes, and legends.
  • Multi-dimensional semantic captions: High-level overviews and detailed analyses of the chart's content and insights.

The framework involves:

  1. Data Collection: Sourcing reliable and diverse datasets (e.g., from Kaggle) with high ratings and performing cleaning.
  2. Automatic Visualization: Using a system like DeepEye to transform CSV data into multiple JSON files, each representing a different visualization scheme (e.g., line charts, bar charts) for a single dataset.
  3. Visualization Mapping: Employing predefined templates (Matplotlib, Vega-Lite, Plotly) to generate actual chart images from the JSON specifications, allowing for diverse visual representations.
  4. Metadata Generation: Extracting low-level analysis tasks (statistical features, trend analysis) and generating chart captions using LLMs like LLaMA 3.1-70B.

This structured metadata then enables a single chart to support multiple downstream tasks, including text-to-chart retrieval, chart summarization, chart-to-table conversion, chart description, and chart question answering.

Datasets

The paper introduces MetaChart, a new large-scale, high-quality dataset built using the ChartCards framework. MetaChart consists of:

  • 10,862 data tables
  • 85,000 raw chart images
  • 170,000 high-quality chart captions
  • 85,000 visualization codes

This dataset is designed to be comprehensive and versatile, supporting multiple downstream chart understanding tasks. The authors also used existing benchmarks for evaluation, such as CRBench for text-to-chart retrieval, and ChartQA and ChartVLM for chart-to-table, chart summary, chart description, and chartQA tasks.

Key Insights and Ideas

  • Unified Metadata for Multi-Task Learning: The core idea is that by generating a rich, unified set of metadata for each chart, a single chart can effectively serve multiple downstream tasks. This moves away from task-specific datasets and towards a more holistic approach.
  • Scalability through Automation: The ChartCards framework automates much of the data generation process, significantly reducing the cost and effort associated with creating large-scale, high-quality chart datasets. This is crucial for advancing research in this area.
  • LLMs as Powerful Caption Generators: The use of LLMs like LLaMA 3.1-70B for generating both chart overviews and detailed analyses demonstrates their capability in creating semantically rich and informative captions, which are vital for training models.
  • MetaChart as a Superior Benchmark: The MetaChart dataset, with its scale, richness of metadata, and quality validation, is positioned as a superior resource for advancing chart understanding research compared to existing datasets.

Key Results

Fine-tuning six state-of-the-art models on the MetaChart dataset resulted in an average performance improvement of 5% across five chart understanding tasks.

  • Text-to-chart retrieval: Significant improvements were observed, with Long-CLIP and Llama 3.2-11B achieving 17% and 28% improvements, respectively.
  • Chart-to-table conversion: Models like Llama 3.2 showed remarkable gains, with Recall improving from 63.17% to 94.92% on the ChartQA-H dataset.
  • ChartQA: LLaVA-NeXT showed dramatic improvements, with an overall increase of 28.68% in accuracy.
  • Crowdsourcing Validation: Approximately 90% of workers rated the generated captions as high quality, indicating good completeness, consistency, diversity, and readability.

Limitations and Future Work

  • Domain Bias: The reliance on Kaggle datasets may introduce domain biases, limiting the generalizability of models to other domains.
  • Misuse Potential: Improved chart understanding technologies could be misused to create misleading visualizations.
  • Human Analytical Skills: Overreliance on automated interpretation might weaken human critical thinking about data visualizations.
  • Task Divergence in ChartQA: The current approach for ChartQA, which uses generated tables as auxiliary input, might introduce task divergence.

Future work could involve expanding the dataset to cover more domains and chart types, addressing potential biases, and exploring more sophisticated methods for chart understanding that better integrate visual and textual information.

Summary updated

BibTeX
@misc{wu2025chartcards,
  author = {Yifan Wu},
  title = {{ChartCards: A Chart-Metadata Generation Framework for Multi-Task Chart Understanding}},
  year = {2025},
  eprint = {2505.15046},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.15046},
}

Loading related content...

Figure