Syntactically Look-Ahead Attention Network for Sentence Compression

Hidetaka Kamigaito, Manabu Okumura

Introduction

Sentence compression aims to shorten sentences while preserving grammatical integrity and core information. While Sequence-to-Sequence (Seq2Seq) models are the standard approach, their unidirectional processing often leads to a "blind spot." Decisions made early in a sentence may inadvertently delete words necessary for supporting information that appears later. To address this, researchers developed the Syntactically Look-Ahead Attention Network (SLAHAN), which utilizes dependency trees to "look ahead" into a sentence's future structure.

The SLAHAN Architecture

SLAHAN enhances the standard Seq2Seq framework with specialized modules designed to capture long-range syntactic dependencies.

The SLAHAN architecture featuring recursive attention and a selective gate mechanism.

As shown in the architecture diagram, the model processes inputs through three primary components:

  1. Recursive Attention: The model tracks both syntactic parents and children. While parent tracking uses a weighted sum, child tracking employs max-pooling to handle cases where a single word has multiple modifiers.
  2. Selective Gate: This component acts as a decision-maker, calculating a gate value to determine whether parent or child information is more relevant for the current decoding step.
  3. Joint Learning: The model is trained to simultaneously compress sentences and predict dependency trees, increasing its robustness against errors from external parsers.

The Importance of Syntactic Children

A key contribution of this research is the emphasis on "child" links (modifiers and dependents). Previous models often focused solely on syntactic heads (parents). However, analysis of sentence structures reveals that as sentences grow longer and more complex, the role of child links becomes increasingly significant.

Ratio of parent vs. child link types across varying sentence lengths.

Data indicates that child links are vital for preserving context. Deleting a child while keeping its parent often results in the loss of specific details that make a summary useful. By using the dependency tree as a map, SLAHAN ensures that "anchor" words for critical phrases are preserved even if they appear far down the line.

Key Results and Insights

SLAHAN established new state-of-the-art benchmarks on the Google Sentence Compression Dataset, achieving an F1 score of 85.5 and high ROUGE scores (R-1: 79.3, R-L: 79.1). The model demonstrated its greatest strength on sentences exceeding 27 words, where standard models typically fail.

Interestingly, the choice of contextualized embeddings (GloVe, ELMo, BERT) showed task-specific sensitivity. While BERT is often the default choice, the authors found that the optimal embedding combination depends heavily on the specific requirements of discourse and named entity preservation in compression tasks.

Conclusion

SLAHAN demonstrates that looking ahead into syntactic structures—specifically tracking child modifiers—significantly improves sentence compression quality. While the selective gate occasionally miscalculates weights in ambiguous contexts, the model provides more informative outputs than traditional baselines. Future research will focus on refining the gating mechanism and integrating these syntactic features into Transformer-based architectures.

Summary updated

Abstract

Sentence compression is the task of compressing a long sentence into a short one by deleting redundant words. In sequence-to-sequence (Seq2Seq) based models, the decoder unidirectionally decides to retain or delete words. Thus, it cannot usually explicitly capture the relationships between decoded words and unseen words that will be decoded in the future time steps. Therefore, to avoid generating ungrammatical sentences, the decoder sometimes drops important words in compressing sentences. To solve this problem, we propose a novel Seq2Seq model, syntactically look-ahead attention network (SLAHAN), that can generate informative summaries by explicitly tracking both dependency parent and child words during decoding and capturing important words that will be decoded in the future. The results of the automatic evaluation on the Google sentence compression dataset showed that SLAHAN achieved the best kept-token-based-F1, ROUGE-1, ROUGE-2 and ROUGE-L scores of 85.5, 79.3, 71.3 and 79.1, respectively. SLAHAN also improved the summarization performance on longer sentences. Furthermore, in the human evaluation, SLAHAN improved informativeness without losing readability.
BibTeX
@inproceedings{kamigaito2020syntactically,
  author = {Hidetaka Kamigaito and Manabu Okumura},
  title = {{Syntactically Look-Ahead Attention Network for Sentence Compression}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6315},
}
Figure