Introduction
Generating a title is a specialized form of summarization where brevity is paramount. Titles typically range from 4 to 10 words, making every word choice critical. Existing sequence-to-sequence (seq2seq) models often struggle with two main issues: static context and the "repetition trap."
Static context refers to fixed source text representations that do not adapt as the title is generated. The repetition trap occurs when standard coverage mechanisms over-penalize useful words in a quest to avoid redundancy, leading to ungrammatical or incomplete titles.
Methodology: The TSE-AC Framework
To address these challenges, the researchers proposed TSE-AC, a framework featuring Timestep-aware Sentence Embedding (TSE) and Acme Coverage (AC).
Timestep-aware Sentence Embedding (TSE)
Unlike traditional models that freeze sentence representations after the encoding phase, TSE dynamically updates sentence embeddings at every decoding step. This allows the model to "re-locate" focus on critical words as the title evolves. The architecture employs a hierarchical encoder using Gated Recurrent Units (GRUs) for sequence modeling and Graph Convolutional Networks (GCNs) to incorporate structural information like dependency trees and lexical similarity between sentences.
Acme Coverage (AC)
Standard "Vanilla Coverage" (VC) tracks attended locations, which can be too restrictive for short titles. Acme Coverage is a more surgical mechanism: it only penalizes the attention weights of words that were actually generated in the output. This ensures that important context remains accessible until it is explicitly used, preventing the model from prematurely "locking out" valuable keywords.
Evaluation and Results
The model was benchmarked on two major datasets: AESLC (Enron email subject lines) and Chinese Gigaword (news headlines).
Performance Metrics
On the AESLC dataset, TSE-AC significantly outperformed baselines. The model achieved a ROUGE-L score of 29.13 on the development set, surpassing both traditional models like PG-net and large-scale pre-trained models like T5.
Human Utility
Beyond automated metrics, human evaluation on the Gigaword dataset revealed that TSE-AC titles were rated as "usable" 80.27% of the time, a substantial improvement over the 68.31% usability score of models using Vanilla Coverage. This highlights the practical benefit of the precision-focused Acme Coverage mechanism in generating informative titles.
Conclusion
TSE-AC demonstrates that treating sentence representation as a moving target and refining coverage mechanisms can yield more informative and concise titles. While the dynamic updates introduce some computational complexity, the benefits in output quality—especially for short-form summarization—are significant. Future work may explore scaling these dynamic embeddings for longer document summarization tasks where the "needle in a haystack" problem is even more pronounced.
Summary updated