Large Language Models (LLMs) have set new benchmarks in abstractive summarization but remain computationally prohibitive for many real-world applications. Conversely, Small Language Models (SLMs) offer efficiency but often struggle with the complex reasoning required for high-quality summarization. This paper presents the LSR-MWF framework, which bridges this gap by distilling structured rationales from LLMs into SLMs. The framework extracts hierarchical "rationales"—comprising essential aspects, supporting sentences, and entity-relation triples—and integrates them into SLMs via a multi-view weak-gated fusion mechanism. Experimental results on CNN/DailyMail and XSum datasets demonstrate that this approach significantly boosts SLM performance, often exceeding the zero-shot capabilities of much larger models.
Introduction
The rise of Large Language Models (LLMs) like GPT-4 and Llama-3 has revolutionized natural language processing, particularly in abstractive summarization. However, their massive parameter counts necessitate significant computational resources, leading to high latency and costs. Small Language Models (SLMs) are more practical for deployment but often produce "coarse-grained" summaries because they fail to capture the underlying logical structure of long documents.
Previous distillation efforts typically focused on the final output summary, treating the LLM as a black box. This paper argues that the "reasoning" process—the intermediate steps an LLM takes to arrive at a summary—is just as valuable. By teaching SLMs to follow a structured path of identifying key aspects and mapping entity relationships, researchers have achieved double-digit performance gains while enhancing model interpretability.
Methodology: The LSR-MWF Framework
The authors propose the LLM-based Structured Rationale-guided Multi-view Weak-gated Fusion (LSR-MWF) framework. This system operates through a three-stage pipeline designed to extract, filter, and integrate high-quality reasoning data.
- Structured Rationale Digging (LSR): Instead of simple prompting, the LLM is tasked with extracting three specific "gems" of information:
- Essential Aspects (EA): The core topics or keywords for each summary sentence.
- Associated Sentences (AS): The specific source text sentences that support those aspects.
- Triple Entity Relations (TER): Structured
[subject # relation # object]triples that map the logical flow. - Best Rationale Selection: To ensure the quality of the distilled data, the framework generates "sub-summaries" based on the extracted rationales. These are then filtered using ROUGE scores against ground-truth summaries to retain only the most accurate reasoning paths.
- Multi-view Weak-gated Fusion (MWF): The SLM architecture is enhanced with three dedicated modules (EA, AS, and TER) that align with the LLM's features using a cosine loss. These features are fused into the SLM's decoder via "weak-gated" units—learnable parameters that adaptively weight each rationale type at different generation stages.
Key Insights and Results
The framework was evaluated on the CNN/DailyMail (CNNDM) and XSum benchmarks. The results were striking: the LSR-MWF model (based on BART-large, ~439M parameters) outperformed the standard BART baseline by 11.2% on CNNDM and 5.8% on XSum (ROUGE-1). Notably, this distilled SLM outperformed the zero-shot performance of much larger models like GPT-3.5 and Llama-3-70B.
A critical finding from the ablation studies was the importance of Triple Entity Relations (TER). Removing relation-level information caused the most significant performance drop, suggesting that understanding how entities relate is more vital for summarization than simply identifying the entities themselves. Furthermore, the "weak gates" revealed that the model's reliance on different rationales is dynamic, often peaking in the final decoder layers where the synthesis of information occurs.
Conclusion and Future Work
LSR-MWF demonstrates that structured rationales can effectively transfer complex reasoning capabilities from LLMs to SLMs. While the current implementation uses standard triples, future research could incorporate more complex linguistic structures, such as anaphora or oppositional relations. Although the initial "digging" phase requires significant compute, the resulting SLM is highly efficient, making it a viable solution for high-performance summarization in resource-constrained environments.
Summary updated