Abstract
Chain-of-Thought (CoT) serves as a critical emerging ability in LLMs, especially when it comes to logical reasoning. Attempts have been made to induce such ability in small models as well by distilling from the data with CoT generated by Large Language Models (LLMs). However, existing methods often simply generate and incorporate more data from LLMs and fail to note the importance of efficiently utilizing existing CoT data. We here propose a new training paradigm AS-ES (Abstractive Segments - Extractive Segments) learning, which exploits the inherent information in CoT for iterative generation. Experiments show that our methods surpass the direct seq2seq training on CoT-extensive tasks like MWP and PET summarization, without data augmentation or altering the model itself. Furthermore, we explore the reason behind the inefficiency of small models in learning CoT and provide an explanation of why AS-ES learning works, giving insights into the underlying mechanism of CoT.
Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities in multi-step reasoning through "Chain-of-Thought" (CoT) prompting. While these abilities can be distilled into smaller language models (SLMs) by training them on LLM-generated rationales, SLMs often struggle to effectively internalize these complex reasoning chains. Standard training paradigms typically treat the entire reasoning sequence as a single target string (direct seq2seq). However, due to limited parameter capacity, small models frequently lose track of the reasoning context or fail to bridge the gap between the initial query and the final conclusion.
The core challenge lies in the fundamental inefficiency of how small models process CoT data. Previous solutions often relied on massive data augmentation or complex multi-model architectures, which are computationally expensive. AS-ES Learning addresses this by decomposing reasoning chains into functional components that align better with the learning dynamics of smaller models.
Methodology: AS-ES Learning
The AS-ES framework deconstructs a reasoning chain into two distinct types of segments:
- Extractive Segments (ES): These segments ground the model by repeating or paraphrasing facts directly from the input context (e.g., "Weng earns $12 an hour").
- Abstractive Segments (AS): These segments represent the actual logical leaps, inferences, or calculations (e.g., "which means she earns $1 per 5 minutes").
The Training Pipeline
The AS-ES learning process involves four main stages:
- Segmentation: CoT data is automatically partitioned into ES and AS. The researchers explored several strategies: Entropy-oriented (splitting based on model uncertainty), Location-oriented (interleaving sentences), and Similarity-oriented (using overlap metrics like BLEU/ROUGE against the query).
- Iterative Generation: Rather than generating the full response in one pass, the model generates one segment at a time. Each generated segment is appended to the input for the next step, creating a "rolling" context that keeps the model anchored.
- Model Architecture: The study compared "Dual-path" (separate models for ES and AS) and "Uni-path" (a single model for both) approaches. The Uni-path model generally performed better, suggesting that learning the transition between extraction and abstraction is a vital part of the reasoning process.
Experimental Setup and Datasets
The method was evaluated on two diverse reasoning domains:
- Math Word Problems (MWP): Utilizing a CoT-augmented version of the GSM8K dataset. This task requires highly sequential, step-by-step logic.
- PET Report Summarization (PET): Introducing cPET-11K, a new dataset of 11,600 Chinese PET/CT report-impression pairs. Unlike math, this requires "parallel" reasoning across multiple anatomical regions.
Key Insights
A significant finding was the "Lower BLEU, Higher Accuracy" Paradox. Models trained with AS-ES often produced lower BLEU scores compared to the ground truth but achieved higher final task accuracy. This indicates that the models are not merely memorizing LLM outputs but are learning to construct their own valid reasoning paths.
Furthermore, the researchers observed that breaking the task into ES and AS lowers the "Loss Boundary." The extractive portions represent an easier learning objective, providing a stable foundation that allows the model to dedicate its limited capacity to the more difficult abstractive steps.
Results and Discussion
AS-ES consistently outperformed standard direct training across models ranging from 77M to 800M parameters:
- Math (MWP): A 77M parameter Flan-T5 model saw accuracy improve from 15.25% to 17.58% using Uni-path interleaving.
- Medical (PET): The Uni-path entropy model improved the BLEU score from 2.17 to 3.83 and significantly reduced the Missing Ratio (MR) of critical anomalies from 39.34% to 24.78%.
Crucially, these improvements were achieved without any additional training data, demonstrating that the structural organization of CoT data is a key lever for performance in small models.
Limitations and Future Work
Despite its success, AS-ES learning faces challenges. The iterative process increases input length, which can hit token limits in long-form tasks like medical reporting. Additionally, the method requires tuning a hyperparameter (beta) for segmentation thresholds. Finally, iterative generation is susceptible to error accumulation; if an early extractive step is incorrect, the subsequent reasoning is likely to fail. Future research may focus on integrating self-correction mechanisms to mitigate these cascading errors.
Summary updated