Visual Question Answering on Scientific Charts Using Fine-Tuned Vision-Language Models

Florian Schleid, Jan Strich, Chris BiemannACL 2025

TL;DR

This paper tackles the challenge of automatically understanding scientific charts, a crucial but difficult task for AI. By fine-tuning large vision-language models (VLMs) like Qwen2.5-VL on scientific chart datasets, the authors achieve state-of-the-art results on the SciVQA shared task, significantly outperforming baselines and demonstrating the power of domain-specific adaptation.

What Problem Does This Paper Address?

Scientific charts are packed with information, often conveying the core findings of research papers. However, automatically extracting this information through Visual Question Answering (VQA) is a significant challenge. Charts involve complex visual elements, spatial reasoning, and numerical understanding. Existing VQA models struggle with the nuances of scientific charts, leading to performance plateaus. This paper aims to improve VQA performance on scientific charts by exploring advanced VLMs and effective fine-tuning strategies.

Methods

The core of this work involves fine-tuning large Vision-Language Models (VLMs), specifically Qwen2.5-VL (available in 7B and 32B parameter variants), for the task of scientific chart VQA. The authors explore several approaches:

  • Zero-Shot and One-Shot Prompting: They investigate how different prompting strategies, including providing clear instructions and a single example (one-shot), influence model performance. This is a standard technique to guide LLMs without explicit training.
  • LoRA Fine-Tuning: They employ Low-Rank Adaptation (LoRA), an efficient fine-tuning technique that significantly reduces computational and memory requirements. This allows for more accessible and faster adaptation of large models.
  • Domain-Specific Fine-Tuning: The primary focus is on fine-tuning the VLMs on the SciVQA dataset, which is specifically curated for scientific chart VQA.
  • Augmenting Training Data: They also experiment with incorporating data from similar domain-specific datasets (SpiQA and ArXivQA) to see if this further boosts performance.

Datasets

  • SciVQA: This is the primary dataset used for training and evaluation. It contains 3,000 real-world chart images from scientific papers, each with seven questions covering multiple-choice, yes/no, unanswerable, and infinite answer set types.
  • SpiQA: Used to augment the training data, this dataset also features scientific charts and primarily contains infinite answer set questions.
  • ArXivQA: Also used for data augmentation, this dataset focuses on multiple-choice questions related to scientific charts.

Key Insights and Ideas

  • Domain-Specific Fine-Tuning is King: The most significant performance gains come from fine-tuning the Qwen2.5-VL models specifically on the SciVQA dataset. This highlights that general-purpose VLMs, while powerful, benefit immensely from targeted adaptation to specialized domains like scientific charts.
  • Model Size Matters (with caveats): The larger Qwen2.5-VL 32B model generally outperforms the 7B variant when fine-tuned. However, in zero-shot settings, the 7B model surprisingly performed better, which the authors attribute to the 32B model generating excessively long answers that penalized the evaluation metrics.
  • Prompting is Important, but Fine-Tuning is Better: While zero-shot and one-shot prompting can yield decent results, they are significantly outperformed by fine-tuning. One-shot prompting, in particular, showed mixed results, working better for the GPT-40 mini baseline than for the Qwen models.
  • Augmenting Data Can Be Tricky: Surprisingly, adding data from SpiQA and ArXivQA to the training set reduced performance compared to fine-tuning solely on SciVQA. This suggests that simply increasing data volume isn't always beneficial and that data quality, domain alignment, and hyperparameter tuning for larger combined datasets are critical.

Key Results

  • The fine-tuned Qwen2.5-VL 32B model achieved a score of 0.836 (average of ROUGE-1, ROUGE-L, and BERTScore), outperforming the GPT-40 mini baseline by 0.206.
  • This fine-tuned model secured 4th place in the SciVQA shared task.
  • Fine-tuning significantly improved performance on multiple-choice and unanswerable questions, suggesting reduced hallucinations and better estimation of when not to answer.
  • A notable observation was the difference in performance between visual and non-visual infinite answer set questions, with visual questions being more challenging.

Limitations and Future Work

  • Visual Infinite Answer Set Questions: The system struggles with visual questions requiring exact numerical answers from charts, often producing approximate but incorrect responses.
  • Generalization Challenges: The performance drop when incorporating external datasets suggests that subtle domain shifts can hinder generalization, even for seemingly related datasets. This points to the need for careful dataset curation and potentially more robust models.
  • Future Work: The authors suggest exploring more sophisticated data selection strategies, manual curation of training data, and extensive hyperparameter tuning for larger combined datasets to further improve performance. They also highlight the need for more research into why augmenting data led to a performance decrease.

Summary updated

BibTeX
@inproceedings{schleid2025visual,
  author = {Florian Schleid and Jan Strich and Chris Biemann},
  title = {{Visual Question Answering on Scientific Charts Using Fine-Tuned Vision-Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure