Introduction
Sentence compression—the task of shortening a sentence while preserving its core meaning—is a fundamental challenge in natural language processing. Traditionally, the most effective models have relied on supervised learning, which requires massive parallel corpora of long sentences paired with human-compressed versions. Such datasets are expensive and difficult to construct.
Existing unsupervised approaches often struggle with two primary issues. First, the "Denoising Trap" occurs when models treat compression as a simple denoising task, often failing to retain critical information. Second, the "Lead Trap" is a common failure mode where models (such as the previous state-of-the-art, SEQ³) simply copy the first half of a sentence and discard the rest, missing salient information located later in the text.
SCAR (Sentence Compression using Autoencoders for Reconstruction) addresses these challenges by using a dual encoder-decoder architecture that learns to identify and remove "inferable" words from raw, unlabeled text.
The SCAR Framework
SCAR utilizes a "Compressor-Reconstructor" architecture designed to evaluate word importance based on its necessity for reconstruction.
- The Compressor (C): This module processes an input sentence and generates an "indicator vector" (a binary mask). For each word, the compressor decides whether to keep it (1) or drop it (0).
- The Reconstructor (R): This module receives only the words selected by the compressor and attempts to regenerate the original, full sentence.
Linkage Loss ($L_{lnk}$)
The core innovation of SCAR is the Linkage Loss. This loss function creates a feedback loop between the two modules by penalizing the network if the compressor drops a word that the reconstructor cannot easily predict, or if it keeps a word that is redundant and easily guessed from context. This forces the model to retain high-information content words (nouns, verbs) while discarding redundant modifiers.
Addressing Stopword Bias
Due to Zipf’s Law, common stopwords (e.g., "the", "is") are statistically easy to reconstruct. Without intervention, the model would drop all grammatical markers, resulting in unreadable "keyword salad." To prevent this, the authors implemented a Stop Predictor ($D_{stop}$) that re-weights the vocabulary distribution, ensuring the model maintains a balance between information density and grammatical structure.
Key Insights and Uniform Coverage
A significant advantage of SCAR is its ability to maintain "summary coverage" across the entire length of the input sentence. Unlike models that rely on position-based heuristics, SCAR evaluates the saliency of words regardless of their position.
As shown in the analysis of input segments, baseline models like LEAD50 and SEQ³ show a sharp decline in information retention after the first 50% of a sentence. In contrast, SCAR maintains a stable overlap percentage, demonstrating that it captures salient information buried deep within the text.
Experimental Results
The model was evaluated on the Gigaword corpus for training and the DUC-2003/2004 benchmarks for testing.
- Performance: SCAR achieved superior ROUGE-1 and ROUGE-L scores compared to previous unsupervised leaders. On DUC-2004, SCAR reached an R-1 score of 22.92, outperforming SEQ³ (22.12).
- User Study: A study with 15 participants revealed that SCAR-compressed text allowed readers to match sentences to headlines 25% faster than reading full text, without a loss in accuracy.
- Correctness: In headline-matching tasks, SCAR-compressed text achieved 66.67% accuracy, significantly higher than DAE (26.67%) and SEQ³ (53.3%).
Conclusion and Future Work
SCAR demonstrates that "inferability" is a powerful proxy for redundancy in natural language. By training a model to minimize reconstruction error, it inherently learns which parts of a sentence are unique and necessary. While the model currently focuses on deletion-based compression, future work may explore abstractive paraphrasing and the integration of SCAR as a preprocessing step for downstream tasks like Machine Translation.
Summary updated