Abstract
Solving complex science questions requires sophisticated Chain-of-Thought (CoT) reasoning that integrates visual evidence with external scientific knowledge. T-SciQ is a novel framework that leverages Large Language Models (LLMs) to generate high-quality "teaching signals" to train smaller multimodal models. By combining standard reasoning with a new plan-based approach (PCoT), T-SciQ achieved a record-breaking 96.18% accuracy on the ScienceQA benchmark, significantly outperforming both human experts and massive models like GPT-4.
Introduction
Science question answering (QA) is a challenging task that requires a multi-step mental process to connect observations with scientific principles. Existing methods typically fine-tune smaller models using human-annotated rationales. However, this approach faces two primary bottlenecks: high collection costs and "knowledge gaps." Human-written explanations are often shallow, focusing only on provided text while omitting the deep world knowledge—such as specific geographic locations or physical laws—essential for robust reasoning.
T-SciQ addresses these issues by using LLMs as "teachers" to provide deeper, more comprehensive reasoning chains that inject essential external knowledge into the training process.
Methods
The T-SciQ framework follows a three-stage pipeline designed to distill the wisdom of a large teacher model (SciTeacher) into a more efficient student model (SciStudent).
1. Generating Teaching Data
Instead of relying on human labels, the framework uses GPT-3.5 to generate two types of rationales:
- QA-CoT: Standard step-by-step explanations.
- QA-PCoT (Plan-based CoT): A structured approach for complex problems involving Lecture Generation (summarizing the required skill), Plan Generation (decomposing the problem), and Rationale Generation (executing the plan).
2. Mixing Teaching Data
Recognizing that different scientific skills require different reasoning depths, the authors employ a skill-based data mixing strategy. They evaluate both CoT and PCoT signals on a validation set and select the superior signal type for each specific skill (e.g., "Reading a map" vs. "Biological classification") to optimize the training set.
3. Fine-Tuning
The student model, a multimodal transformer, is trained in two stages. It first learns to generate the rationale (the "why") and then uses that generated rationale alongside the original question and image to infer the final answer.
Experimental Results
The framework was primarily evaluated on ScienceQA, a benchmark of 21,208 multimodal questions. T-SciQ achieved a state-of-the-art 96.18% accuracy, surpassing the human average of 88.40%. Remarkably, the 738M parameter student model outperformed much larger models, including GPT-4 (82.69%) and Chameleon (86.54%), on this benchmark. It also showed a 4.5% improvement over the previous best fine-tuned baseline, Multimodal-CoT.
To demonstrate versatility, the authors also tested the method on general reasoning benchmarks such as Aqua (arithmetic), CommonSenseQA, and StrategyQA, where it continued to show robust performance.
Discussion and Conclusion
T-SciQ demonstrates that LLM-generated rationales can be superior to human annotations for training because they explicitly include the external knowledge humans often take for granted. The success of the PCoT approach highlights that decomposition and planning are vital for complex scientific reasoning.
While highly effective, the framework's performance is tied to the quality of the teacher LLM; inaccuracies or hallucinations in the teacher's output can propagate to the student. Future work will explore parameter-efficient fine-tuning (PEFT) to further reduce compute requirements and expand the framework to broader multimodal tasks.
Summary updated