Abstract
Chain-of-Thought (CoT) serves as a critical emerging ability in LLMs, especially when it comes to logical reasoning. Attempts have been made to induce such ability in small models as well by distilling from the data with CoT generated by Large Language Models (LLMs). However, existing methods often simply generate and incorporate more data from LLMs and fail to note the importance of efficiently utilizing existing CoT data. We here propose a new training paradigm AS-ES (Abstractive Segments - Extractive Segments) learning, which exploits the inherent information in CoT for iterative generation. Experiments show that our methods surpass the direct seq2seq training on CoT-extensive tasks like MWP and PET summarization, without data augmentation or altering the model itself. Furthermore, we explore the reason behind the inefficiency of small models in learning CoT and provide an explanation of why AS-ES learning works, giving insights into the underlying mechanism of CoT.
Introduction
Large Language Models (LLMs) have demonstrated remarkable reasoning capabilities through Chain-of-Thought (CoT) prompting. While distilling these abilities into Small Language Models (SLMs) is a common goal, standard sequence-to-sequence (seq2seq) distillation often falls short. The primary challenge lies in the dual cognitive load placed on SLMs: they must simultaneously maintain context (extraction) and derive new logical steps (abstraction).
To address this, we introduce AS-ES Learning, a paradigm that decomposes CoT data into two functional components: Extractive Segments (ES), which anchor the reasoning in the provided context, and Abstractive Segments (AS), which represent the logical deductions.
Methodology: The AS-ES Framework
The core of AS-ES learning is the transformation of a monolithic reasoning string into an iterative, segment-based generation process.
Segmentation and Labeling
The workflow begins by splitting the original CoT answer into sentences based on punctuation. Each sentence is then assigned a score ($M$) based on its similarity to the input or its information entropy. Sentences with low scores (high similarity to context) are labeled as Extractive Segments (ES), while those with high scores (new information) are labeled as Abstractive Segments (AS).
Iterative Training
Instead of a single pass, the model is trained to generate segments sequentially. Each generated segment is appended to the input for the subsequent step, effectively providing the model with a "working memory" of its own reasoning path. We explored two architectures:
- Dual-path (DSM): Utilizing separate models for extraction and abstraction.
- Uni-path (USM): A single model trained on the interleaved segments.
Experimental Setup
We evaluated AS-ES on two distinct reasoning domains:
- Math Word Problems (MWP): Using a CoT-augmented version of GSM8K.
- PET-scan Summarization (PET): Introducing cPET-11K, a dataset of 11,600 Chinese PET/CT report-impression pairs requiring complex diagnostic synthesis.
Results and Discussion
Performance Gains
On MWP tasks, the AS-ES Uni-path model (based on Flan-T5) showed consistent improvements. For a 77M parameter model, accuracy rose from 15.25% to 17.58%. For the 250M model, it rose from 19.23% to 21.70%. In the medical domain, the method improved BLEU scores by 50.45% and significantly reduced the "Missing Ratio" of anomalies from 39.34 to 24.78.
The Loss Boundary Theory
A key finding is that AS-ES learning effectively lowers the "loss boundary." By breaking the complex CoT into smaller segments, the training loss for each step is lower than that of a single long sequence. This suggests that SLMs are not inherently incapable of reasoning; rather, they require data structures that align with their limited per-step computational capacity.
Qualitative Insights
Interestingly, in math tasks, AS-ES models sometimes exhibited lower BLEU scores compared to baselines while achieving higher final answer accuracy. This suggests the models are learning the underlying logic rather than merely mimicking the teacher's surface-level phrasing.
Conclusion
AS-ES learning provides a data-efficient pathway for inducing CoT abilities in small models without requiring additional LLM-generated data. By aligning the training objective with the iterative nature of human reasoning, we enable SLMs to perform complex tasks more effectively. Future work will investigate the scalability of this approach to multi-modal reasoning and its integration with reinforcement learning.
Summary updated