SCAR: Sentence Compression using Autoencoders for Reconstruction

Chanakya Malireddy, Tirth Maniar, Manish Shrivastava

Introduction

Sentence compression—the task of shortening a sentence while preserving its core meaning—is a fundamental challenge in natural language processing. Traditionally, the most effective models have relied on supervised learning, which requires massive parallel corpora of long sentences paired with human-compressed versions. Such datasets are expensive and difficult to construct.

Existing unsupervised approaches often struggle with two primary issues. First, the "Denoising Trap" occurs when models treat compression as a simple denoising task, often failing to retain critical information. Second, the "Lead Trap" is a common failure mode where models (such as the previous state-of-the-art, SEQ³) simply copy the first half of a sentence and discard the rest, missing salient information located later in the text.

SCAR (Sentence Compression using Autoencoders for Reconstruction) addresses these challenges by using a dual encoder-decoder architecture that learns to identify and remove "inferable" words from raw, unlabeled text.

The SCAR Framework

SCAR utilizes a "Compressor-Reconstructor" architecture designed to evaluate word importance based on its necessity for reconstruction.

The SCAR architecture featuring the Compressor (C) and Reconstructor (R) modules, including the Indicator Extractor and Stop Predictor.
  1. The Compressor (C): This module processes an input sentence and generates an "indicator vector" (a binary mask). For each word, the compressor decides whether to keep it (1) or drop it (0).
  2. The Reconstructor (R): This module receives only the words selected by the compressor and attempts to regenerate the original, full sentence.

Linkage Loss ($L_{lnk}$)

The core innovation of SCAR is the Linkage Loss. This loss function creates a feedback loop between the two modules by penalizing the network if the compressor drops a word that the reconstructor cannot easily predict, or if it keeps a word that is redundant and easily guessed from context. This forces the model to retain high-information content words (nouns, verbs) while discarding redundant modifiers.

Addressing Stopword Bias

Due to Zipf’s Law, common stopwords (e.g., "the", "is") are statistically easy to reconstruct. Without intervention, the model would drop all grammatical markers, resulting in unreadable "keyword salad." To prevent this, the authors implemented a Stop Predictor ($D_{stop}$) that re-weights the vocabulary distribution, ensuring the model maintains a balance between information density and grammatical structure.

Key Insights and Uniform Coverage

A significant advantage of SCAR is its ability to maintain "summary coverage" across the entire length of the input sentence. Unlike models that rely on position-based heuristics, SCAR evaluates the saliency of words regardless of their position.

Comparison of overlap percentage across sentence segments, showing SCAR's consistent performance compared to the 'Lead Trap' seen in SEQ³ and LEAD50.

As shown in the analysis of input segments, baseline models like LEAD50 and SEQ³ show a sharp decline in information retention after the first 50% of a sentence. In contrast, SCAR maintains a stable overlap percentage, demonstrating that it captures salient information buried deep within the text.

Experimental Results

The model was evaluated on the Gigaword corpus for training and the DUC-2003/2004 benchmarks for testing.

  • Performance: SCAR achieved superior ROUGE-1 and ROUGE-L scores compared to previous unsupervised leaders. On DUC-2004, SCAR reached an R-1 score of 22.92, outperforming SEQ³ (22.12).
  • User Study: A study with 15 participants revealed that SCAR-compressed text allowed readers to match sentences to headlines 25% faster than reading full text, without a loss in accuracy.
  • Correctness: In headline-matching tasks, SCAR-compressed text achieved 66.67% accuracy, significantly higher than DAE (26.67%) and SEQ³ (53.3%).

Conclusion and Future Work

SCAR demonstrates that "inferability" is a powerful proxy for redundancy in natural language. By training a model to minimize reconstruction error, it inherently learns which parts of a sentence are unique and necessary. While the model currently focuses on deletion-based compression, future work may explore abstractive paraphrasing and the integration of SCAR as a preprocessing step for downstream tasks like Machine Translation.

Summary updated

Abstract

Sentence compression is the task of shortening a sentence while retaining its meaning. Most methods proposed for this task rely on labeled or paired corpora (containing pairs of verbose and compressed sentences), which is often expensive to collect. To overcome this limitation, we present a novel unsupervised deep learning framework (SCAR) for deletion-based sentence compression. SCAR is primarily composed of two encoder-decoder pairs: a compressor and a reconstructor. The compressor masks the input, and the reconstructor tries to regenerate it. The model is entirely trained on unlabeled data and does not require additional inputs such as explicit syntactic information or optimal compression length. SCAR’s merit lies in the novel Linkage Loss function, which correlates the compressor and its effect on reconstruction, guiding it to drop inferable tokens. SCAR achieves higher ROUGE scores on benchmark datasets than the existing state-of-the-art methods and baselines. We also conduct a user study to demonstrate the application of our model as a text highlighting system. Using our model to underscore salient information facilitates speed-reading and reduces the time required to skim a document.
BibTeX
@inproceedings{malireddy2020scar,
  author = {Chanakya Malireddy and Tirth Maniar and Manish Shrivastava},
  title = {{SCAR: Sentence Compression using Autoencoders for Reconstruction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-srw.13},
}
Figure