A Timestep aware Sentence Embedding and Acme Coverage for Brief but Informative Title Generation

Quanbin Wang, Xiexiong Lin, Feng WangNAACL 2022 · DOI

Introduction

Generating a title is a specialized form of summarization where brevity is paramount. Titles typically range from 4 to 10 words, making every word choice critical. Existing sequence-to-sequence (seq2seq) models often struggle with two main issues: static context and the "repetition trap."

Static context refers to fixed source text representations that do not adapt as the title is generated. The repetition trap occurs when standard coverage mechanisms over-penalize useful words in a quest to avoid redundancy, leading to ungrammatical or incomplete titles.

Methodology: The TSE-AC Framework

To address these challenges, the researchers proposed TSE-AC, a framework featuring Timestep-aware Sentence Embedding (TSE) and Acme Coverage (AC).

Timestep-aware Sentence Embedding (TSE)

Unlike traditional models that freeze sentence representations after the encoding phase, TSE dynamically updates sentence embeddings at every decoding step. This allows the model to "re-locate" focus on critical words as the title evolves. The architecture employs a hierarchical encoder using Gated Recurrent Units (GRUs) for sequence modeling and Graph Convolutional Networks (GCNs) to incorporate structural information like dependency trees and lexical similarity between sentences.

Architecture of the TSE-AC framework showing the hierarchical encoder and dynamic update mechanism.

Acme Coverage (AC)

Standard "Vanilla Coverage" (VC) tracks attended locations, which can be too restrictive for short titles. Acme Coverage is a more surgical mechanism: it only penalizes the attention weights of words that were actually generated in the output. This ensures that important context remains accessible until it is explicitly used, preventing the model from prematurely "locking out" valuable keywords.

Evaluation and Results

The model was benchmarked on two major datasets: AESLC (Enron email subject lines) and Chinese Gigaword (news headlines).

Performance Metrics

On the AESLC dataset, TSE-AC significantly outperformed baselines. The model achieved a ROUGE-L score of 29.13 on the development set, surpassing both traditional models like PG-net and large-scale pre-trained models like T5.

Comparison of TSE-AC against state-of-the-art baselines on the AESLC dataset.

Human Utility

Beyond automated metrics, human evaluation on the Gigaword dataset revealed that TSE-AC titles were rated as "usable" 80.27% of the time, a substantial improvement over the 68.31% usability score of models using Vanilla Coverage. This highlights the practical benefit of the precision-focused Acme Coverage mechanism in generating informative titles.

Conclusion

TSE-AC demonstrates that treating sentence representation as a moving target and refining coverage mechanisms can yield more informative and concise titles. While the dynamic updates introduce some computational complexity, the benefits in output quality—especially for short-form summarization—are significant. Future work may explore scaling these dynamic embeddings for longer document summarization tasks where the "needle in a haystack" problem is even more pronounced.

Summary updated

BibTeX
@inproceedings{wang2022timestep,
  author = {Quanbin Wang and Xiexiong Lin and Feng Wang},
  title = {{A Timestep aware Sentence Embedding and Acme Coverage for Brief but Informative Title Generation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-naacl.146},
}
Figure