Improving Long Text Understanding with Knowledge Distilled from Summarization Model

Yan Liu, Yazheng Yang, Xiaokang Chen

Abstract

Understanding long documents remains a significant challenge in natural language processing due to the inherent noise and computational complexity of processing extensive sequences. We propose the Gist Detector, a lightweight student model that learns to identify salient information by distilling knowledge from a large-scale abstractive summarization teacher. By mimicking the teacher's attention patterns, the Gist Detector produces a saliency map that emphasizes the "gist" of a document. Integrating these importance weights into downstream architectures significantly improves performance across document classification, open-domain question answering, and text style transfer while maintaining high efficiency.

Introduction

Long text understanding is often a "needle in a haystack" problem. While humans can quickly skim a report to find core arguments, standard NLP models frequently struggle with redundant information and irrelevant "noise." Transformer-based models face additional hurdles due to the quadratic complexity of self-attention, making the processing of long documents computationally expensive.

While abstractive summarization models are inherently designed to find the "gist" of a text, their large size and auto-regressive nature make them impractical for use as general-purpose pre-processors. This research introduces a method to extract the saliency-detecting capabilities of these large models and transfer them into a compact, non-autoregressive encoder called the Gist Detector.

Methods

The proposed framework utilizes a Teacher-Student distillation approach to create an efficient saliency mapper.

  1. The Teacher Model: A large-scale abstractive summarization model (Transformer encoder-decoder) is trained on the CNN/Daily Mail dataset to identify key information necessary for generating summaries.
  2. The Student (Gist Detector): A lightweight 4-layer Transformer encoder is trained to predict the teacher's attention distribution. The "soft target" for the student is the geometric mean of the teacher's attention across all decoding steps, representing a global saliency map.
  3. Integration: The Gist Detector generates a probability distribution ($p$) over the input tokens. This distribution is fused with the representations of downstream models (e.g., BiLSTM or QA selectors) using a tunable hyperparameter to prioritize salient content and suppress noise.
The architecture of the Gist Detector framework, showing the distillation from the Teacher (Summarization) to the Student (Gist Detector) and its application to a Downstream Model.

Experimental Results

The Gist Detector (GD) was evaluated across several diverse benchmarks, demonstrating consistent improvements over strong baselines.

Document Classification

On the FDU-MTL dataset, which spans 16 domains including product reviews and IMDB data, the addition of the Gist Detector provided a substantial boost. A standard BiLSTM baseline achieved an overall accuracy of 82.4%, which increased to 88.2% when augmented with GD. This suggests that the "importance" patterns learned from news data generalize effectively to other domains.

Performance comparison on the FDU-MTL classification dataset across 16 domains. The BiLSTM+GD model consistently outperforms the baseline BiLSTM and other multi-task learning methods.

Open-Domain Question Answering

In QA tasks using TriviaQA and SearchQA, the Gist Detector helped the "selector" module filter out irrelevant passages more precisely. On TriviaQA, the OpenQA baseline improved from 48.7 to 50.3 Exact Match (EM). The model also showed significant gains in retrieval metrics, such as Hit@1 scores.

Text Style Transfer

For non-parallel style transfer on Amazon and Yelp datasets, the GD-augmented model (ARAE + GD) achieved higher human evaluation scores for content correlation and fluency. By focusing on the "gist," the model better preserved the original meaning of the text during the style transformation process.

Conclusion

The Gist Detector offers a computationally efficient way to leverage the power of large summarization models for general long-text understanding. By treating attention as a proxy for saliency and distilling it into a small encoder, the researchers successfully improved performance across multiple NLP tasks. Future work may explore more sophisticated fusion mechanisms and the application of this distillation technique to specialized long-sequence architectures like the Longformer.

Summary updated

Abstract

Long text understanding is important yet challenging for natural language processing. A long article or document usually contains many redundant words that are not pertinent to its gist and sometimes can be regarded as noise. With recent advances of abstractive summarization, we propose our Gist Detector to leverage the gist detection ability of a summarization model and integrate the extracted gist into downstream models to enhance their long text understanding ability. Specifically, Gist Detector first learns the gist detection knowledge distilled from a summarization model, and then produces gist-aware representations to augment downstream models. We evaluate our method on three different tasks: long document classification, distantly supervised open-domain question answering, and non-parallel text style transfer. The experimental results show that our method can significantly improve the performance of baseline models on all tasks.
BibTeX
@inproceedings{liu2024improving,
  author = {Yan Liu and Yazheng Yang and Xiaokang Chen},
  title = {{Improving Long Text Understanding with Knowledge Distilled from Summarization Model}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=gYkDLd6W3k},
}
Figure