Abstract
Understanding long documents remains a significant challenge in natural language processing due to the inherent noise and computational complexity of processing extensive sequences. We propose the Gist Detector, a lightweight student model that learns to identify salient information by distilling knowledge from a large-scale abstractive summarization teacher. By mimicking the teacher's attention patterns, the Gist Detector produces a saliency map that emphasizes the "gist" of a document. Integrating these importance weights into downstream architectures significantly improves performance across document classification, open-domain question answering, and text style transfer while maintaining high efficiency.
Introduction
Long text understanding is often a "needle in a haystack" problem. While humans can quickly skim a report to find core arguments, standard NLP models frequently struggle with redundant information and irrelevant "noise." Transformer-based models face additional hurdles due to the quadratic complexity of self-attention, making the processing of long documents computationally expensive.
While abstractive summarization models are inherently designed to find the "gist" of a text, their large size and auto-regressive nature make them impractical for use as general-purpose pre-processors. This research introduces a method to extract the saliency-detecting capabilities of these large models and transfer them into a compact, non-autoregressive encoder called the Gist Detector.
Methods
The proposed framework utilizes a Teacher-Student distillation approach to create an efficient saliency mapper.
- The Teacher Model: A large-scale abstractive summarization model (Transformer encoder-decoder) is trained on the CNN/Daily Mail dataset to identify key information necessary for generating summaries.
- The Student (Gist Detector): A lightweight 4-layer Transformer encoder is trained to predict the teacher's attention distribution. The "soft target" for the student is the geometric mean of the teacher's attention across all decoding steps, representing a global saliency map.
- Integration: The Gist Detector generates a probability distribution ($p$) over the input tokens. This distribution is fused with the representations of downstream models (e.g., BiLSTM or QA selectors) using a tunable hyperparameter to prioritize salient content and suppress noise.
Experimental Results
The Gist Detector (GD) was evaluated across several diverse benchmarks, demonstrating consistent improvements over strong baselines.
Document Classification
On the FDU-MTL dataset, which spans 16 domains including product reviews and IMDB data, the addition of the Gist Detector provided a substantial boost. A standard BiLSTM baseline achieved an overall accuracy of 82.4%, which increased to 88.2% when augmented with GD. This suggests that the "importance" patterns learned from news data generalize effectively to other domains.
Open-Domain Question Answering
In QA tasks using TriviaQA and SearchQA, the Gist Detector helped the "selector" module filter out irrelevant passages more precisely. On TriviaQA, the OpenQA baseline improved from 48.7 to 50.3 Exact Match (EM). The model also showed significant gains in retrieval metrics, such as Hit@1 scores.
Text Style Transfer
For non-parallel style transfer on Amazon and Yelp datasets, the GD-augmented model (ARAE + GD) achieved higher human evaluation scores for content correlation and fluency. By focusing on the "gist," the model better preserved the original meaning of the text during the style transformation process.
Conclusion
The Gist Detector offers a computationally efficient way to leverage the power of large summarization models for general long-text understanding. By treating attention as a proxy for saliency and distilling it into a small encoder, the researchers successfully improved performance across multiple NLP tasks. Future work may explore more sophisticated fusion mechanisms and the application of this distillation technique to specialized long-sequence architectures like the Longformer.
Summary updated