RefChartQA: Grounding Visual Answer on Chart Images Through Instruction Tuning

Alexander Vogel, Omar Moured, Yufan Chen, Jiaming Zhang, Rainer Stiefelhagen

Abstract

While Vision-Language Models (VLMs) have demonstrated impressive capabilities in interpreting data visualizations, they often operate as "black boxes," providing answers without explicit reference to the underlying visual evidence. This paper introduces RefChartQA, a novel benchmark designed to incorporate Visual Answer Grounding (VAG) into chart question answering. By instruction-tuning models to localize relevant chart elements using bounding boxes, the authors demonstrate that spatial awareness not only enhances interpretability but also improves response accuracy by up to 15% while significantly reducing hallucinations.

Introduction

In the realm of chart question answering (CQA), current Multimodal Large Language Models (MLLMs) frequently struggle with transparency. When a model identifies the highest value in a bar chart, it rarely provides the spatial coordinates of the data point it used to reach that conclusion. This lack of grounding makes it difficult for users to verify the model's logic, particularly in tasks involving complex arithmetic or multi-step reasoning.

Applying visual grounding to charts is inherently more complex than grounding natural images. Charts contain an interleaved mix of text, numerical data, and geometric shapes (bars, lines, points). General-purpose models often lack the specialized spatial-semantic understanding required to link a textual answer back to specific visual components like axes, legends, or data markers. RefChartQA addresses this gap by providing a large-scale dataset and a framework for training models to "point" to their evidence.

The RefChartQA Benchmark

RefChartQA is built upon the existing ChartQA and ChartQA-PoT datasets, contributing 73,702 image-question-grounding pairs. The dataset is split into training, validation, and test sets, covering both machine-generated queries and human-authored questions.

Automated Annotation Pipeline

To overcome the prohibitive cost of manual bounding-box annotation, the authors developed a three-stage automated pipeline to generate high-quality grounding data from chart metadata:

  1. Single-Element Extraction: For straightforward data-retrieval questions, the pipeline uses heuristics and string matching to identify the relevant chart element.
  2. Program-of-Thought (PoT) Extraction: For arithmetic queries, the pipeline utilizes executable Python code to trace logical steps back to specific visual elements.
  3. GPT-based Annotation: For the most complex reasoning tasks, GPT-4o-mini analyzes chart metadata to identify the specific elements necessary to formulate an answer.

Methodology: Instruction Tuning

The authors fine-tuned six state-of-the-art MLLMs—including Qwen-VL, TinyChart, and DocOwl—using a pixel-to-sequence approach. Models were trained to output bounding box coordinates as text tokens immediately preceding the final answer (e.g., [box coordinates] | answer). To maintain efficiency, most models were adapted using Low-Rank Adaptation (LoRA).

Key Insights and Results

Grounding as a Hallucination Guardrail

A significant finding of the study is that grounding acts as a cognitive anchor. Without grounding, models often attempt unnecessary and complex mathematical reasoning for simple retrieval questions, leading to errors. Forcing the model to localize the data first streamlines its attention, bypassing redundant reasoning loops and reducing the likelihood of hallucinations.

Architectural Importance

The study revealed that model size is not the sole determinant of spatial awareness. For instance, the TinyChart-3B model often performed as well as or better than the much larger DocOwl 1.5-8B. This suggests that specific architectural features, such as TinyChart’s Visual Token Merging module—which aggregates similar visual tokens—are more effective at preserving spatial knowledge than sheer parameter count.

Performance Metrics

  • Accuracy Gains: Incorporating the grounding task improved the "relaxed accuracy" of final answers by up to 15% across several models, with the most notable gains occurring in complex human-posed and PoT splits.
  • Grounding Success: On simple machine-generated queries, models achieved a 70–84% success rate (measured by P@FI, requiring all groundings for a question to be correct).
  • The Complexity Cliff: A significant performance gap remains for complex queries. On human-posed questions requiring multi-element reasoning (e.g., averaging multiple data points), grounding performance dropped to 22–50%, indicating that current models still struggle with high-dimensional spatial-logical interdependencies.

Limitations and Future Work

While RefChartQA represents a major step forward, the authors note that bounding boxes may not be the ideal format for all chart elements. Future research may explore semantic segmentation or keypoint detection to capture finer structural details. Additionally, the automated pipeline can occasionally inherit errors from source datasets or struggle with ambiguous grounding (e.g., whether to point to a legend entry or the data bar itself).

Future directions include developing adaptive multi-grounding paradigms that can switch between boxes, points, and masks depending on the query context, as well as deeper investigations into how different VLM architectures process spatial signals.

Summary updated

BibTeX
@inproceedings{vogel2025refchartqa,
  author = {Alexander Vogel and Omar Moured and Yufan Chen and Jiaming Zhang and Rainer Stiefelhagen},
  title = {{RefChartQA: Grounding Visual Answer on Chart Images Through Instruction Tuning}},
  booktitle = {ICDAR},
  year = {2025},
  doi = {10.1007/978-3-032-04627-7\_30},
}
Figure