T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Large Language Model Signals for Science Question Answering

Lei Wang, Yi Hu, Jiabang He, Xing Xu, Ning Liu, Hui Liu, Heng Tao Shen

Abstract

Solving complex science questions requires sophisticated Chain-of-Thought (CoT) reasoning that integrates visual evidence with external scientific knowledge. T-SciQ is a novel framework that leverages Large Language Models (LLMs) to generate high-quality "teaching signals" to train smaller multimodal models. By combining standard reasoning with a new plan-based approach (PCoT), T-SciQ achieved a record-breaking 96.18% accuracy on the ScienceQA benchmark, significantly outperforming both human experts and massive models like GPT-4.

Introduction

Science question answering (QA) is a challenging task that requires a multi-step mental process to connect observations with scientific principles. Existing methods typically fine-tune smaller models using human-annotated rationales. However, this approach faces two primary bottlenecks: high collection costs and "knowledge gaps." Human-written explanations are often shallow, focusing only on provided text while omitting the deep world knowledge—such as specific geographic locations or physical laws—essential for robust reasoning.

Comparison of human-annotated and LLM-generated reasoning signals in ScienceQA.

T-SciQ addresses these issues by using LLMs as "teachers" to provide deeper, more comprehensive reasoning chains that inject essential external knowledge into the training process.

Methods

The T-SciQ framework follows a three-stage pipeline designed to distill the wisdom of a large teacher model (SciTeacher) into a more efficient student model (SciStudent).

1. Generating Teaching Data

Instead of relying on human labels, the framework uses GPT-3.5 to generate two types of rationales:

  • QA-CoT: Standard step-by-step explanations.
  • QA-PCoT (Plan-based CoT): A structured approach for complex problems involving Lecture Generation (summarizing the required skill), Plan Generation (decomposing the problem), and Rationale Generation (executing the plan).

2. Mixing Teaching Data

Recognizing that different scientific skills require different reasoning depths, the authors employ a skill-based data mixing strategy. They evaluate both CoT and PCoT signals on a validation set and select the superior signal type for each specific skill (e.g., "Reading a map" vs. "Biological classification") to optimize the training set.

3. Fine-Tuning

The student model, a multimodal transformer, is trained in two stages. It first learns to generate the rationale (the "why") and then uses that generated rationale alongside the original question and image to infer the final answer.

The three-stage T-SciQ pipeline: data generation, mixing, and fine-tuning.

Experimental Results

The framework was primarily evaluated on ScienceQA, a benchmark of 21,208 multimodal questions. T-SciQ achieved a state-of-the-art 96.18% accuracy, surpassing the human average of 88.40%. Remarkably, the 738M parameter student model outperformed much larger models, including GPT-4 (82.69%) and Chameleon (86.54%), on this benchmark. It also showed a 4.5% improvement over the previous best fine-tuned baseline, Multimodal-CoT.

To demonstrate versatility, the authors also tested the method on general reasoning benchmarks such as Aqua (arithmetic), CommonSenseQA, and StrategyQA, where it continued to show robust performance.

Discussion and Conclusion

T-SciQ demonstrates that LLM-generated rationales can be superior to human annotations for training because they explicitly include the external knowledge humans often take for granted. The success of the PCoT approach highlights that decomposition and planning are vital for complex scientific reasoning.

While highly effective, the framework's performance is tied to the quality of the teacher LLM; inaccuracies or hallucinations in the teacher's output can propagate to the student. Future work will explore parameter-efficient fine-tuning (PEFT) to further reduce compute requirements and expand the framework to broader multimodal tasks.

Summary updated

Abstract

Large Language Models (LLMs) have recently demonstrated exceptional performance in various Natural Language Processing (NLP) tasks. They have also shown the ability to perform chain-of-thought (CoT) reasoning to solve complex problems. Recent studies have explored CoT reasoning in complex multimodal scenarios, such as the science question answering task, by fine-tuning multimodal models with high-quality human-annotated CoT rationales. However, collecting high-quality COT rationales is usually time-consuming and costly. Besides, the annotated rationales are hardly accurate due to the external essential information missed. To address these issues, we propose a novel method termed T-SciQ that aims at teaching science question answering with LLM signals. The T-SciQ approach generates high-quality CoT rationales as teaching signals and is advanced to train much smaller models to perform CoT reasoning in complex modalities. Additionally, we introduce a novel data mixing strategy to produce more effective teaching data samples for simple and complex science question answer problems. Extensive experimental results show that our T-SciQ method achieves a new state-of-the-art performance on the ScienceQA benchmark, with an accuracy of 96.18%. Moreover, our approach outperforms the most powerful fine-tuned baseline by 4.5%. The code is publicly available at https://github.com/T-SciQ/T-SciQ.
BibTeX
@inproceedings{wang2024t,
  author = {Lei Wang and Yi Hu and Jiabang He and Xing Xu and Ning Liu and Hui Liu and Heng Tao Shen},
  title = {{T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Large Language Model Signals for Science Question Answering}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i17.29884},
}
Figure