STRASS: A Light and Effective Method for Extractive Summarization Based on Sentence Embeddings

Léo Bouscarrat, Antoine Bonnefoy, Thomas Peel, Cécile PereiraACL 2019 · DOI

Introduction

Automatic text summarization is traditionally divided into abstractive and extractive approaches. Abstractive models, such as Pointer-Generator Networks (PGN), generate new sentences but are computationally intensive and prone to "hallucinations." Extractive methods select existing sentences, offering better factual consistency but often lacking semantic sophistication. STRASS (Summarization by TRAnsformation Selection and Scoring) addresses these challenges by leveraging sentence embeddings in a lightweight, CPU-trainable architecture that maintains high semantic awareness.

Methodology: The STRASS Pipeline

The core innovation of STRASS is a simple neural transformation that aligns document embeddings with their ideal summary counterparts. The process follows four distinct stages:

  1. Transformation: A document embedding is passed through a single fully-connected dense layer. This layer learns an affine function to "steer" the document's representation toward the semantic space of a human-written summary.
  2. Sentence Selection: The model calculates cosine similarity between the transformed document embedding and individual sentence embeddings. A differentiable sigmoid function and threshold determine sentence inclusion.
  3. Approximation: During training, the model approximates the embedding of the generated summary via a weighted average of selected sentence embeddings.
  4. Scoring & Loss: Optimization balances semantic similarity to the ground-truth summary against a compression ratio to ensure conciseness.
The four-stage STRASS architecture: Transformation, Selection, Approximation, and Scoring.

Experimental Evaluation

The researchers evaluated STRASS on two distinct benchmarks:

  • CNN/DailyMail: A standard English news dataset containing over 300,000 articles.
  • CASS: A novel French legal dataset comprising 129,445 judgments from the French Court of Cassation. Legal text presents a unique challenge due to its formal structure and specialized vocabulary.

Results and Efficiency

STRASS demonstrates exceptional performance in specialized domains. On the CASS dataset, it achieved a ROUGE-1 score of 52.68, nearly matching the performance of the abstractive PGN (53.25) and the more complex rnn-ext (53.05).

A primary advantage of STRASS is its computational efficiency. Unlike transformer-based models requiring GPU clusters, STRASS can be trained and run on a standard CPU. It processed over 13,000 summaries for the CASS test set in under three minutes on an Intel i7-8550U. As shown in the figure below, the processing time scales linearly with the number of sentences, making it highly scalable for long documents.

Processing time of STRASS scales linearly with the number of sentences in a document.

Discussion and Limitations

While highly effective for legal documents, STRASS lagged behind state-of-the-art models on the CNN/DailyMail news dataset (33.99 vs. 40.17 ROUGE-1). This discrepancy is attributed to two factors:

  1. Multi-Topic Documents: Relying on a single transformed embedding can cause the model to miss sub-topics in long, diverse articles.
  2. Positional Bias: News summaries often favor the first few sentences (Lead-3 bias). STRASS is position-agnostic, which disadvantages it in news contexts compared to models that prioritize the document's beginning.

Conclusion

STRASS proves that massive architectures are not always necessary for effective summarization. By using a simple linear transformation to "steer" semantic representations, STRASS offers a fast, CPU-friendly alternative for extractive summarization, particularly in specialized domains like law where summaries are highly focused and extractive in nature.

Summary updated

BibTeX
@inproceedings{bouscarrat2019strass,
  author = {Léo Bouscarrat and Antoine Bonnefoy and Thomas Peel and Cécile Pereira},
  title = {{STRASS: A Light and Effective Method for Extractive Summarization Based on Sentence Embeddings}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-2034},
}
Figure