TL;DR
This paper introduces a novel evaluation metric, STAS, for topic-controllable summarization and proposes efficient control token methods for Transformer-based summarization models. Their approach significantly outperforms existing methods in both summary quality and inference speed, even for unseen topics.
What Problem Does This Paper Address?
Generating summaries that focus on specific user interests, known as topic-controllable summarization, has many applications. However, current methods face two main challenges: 1) they often rely on older recurrent architectures that are slower and less performant than modern Transformers, and 2) there's a lack of a reliable automatic evaluation metric to assess how well a summary adheres to a given topic. Existing methods either use generic metrics like ROUGE (which don't capture topical focus) or require expensive human evaluations. This paper aims to bridge this gap by proposing a new evaluation metric and developing more efficient and effective topic-control mechanisms for state-of-the-art Transformer models.
Methods
The authors propose two main contributions:
- Summarization Topic Affinity Score (STAS): This is a novel metric designed to quantitatively evaluate topic-controllable summaries. It works by representing both topics and generated summaries as vectors (initially using TF-IDF, but adaptable to other methods like BERT). STAS then calculates the cosine similarity between a summary's vector and the vector of the desired topic, normalized by the maximum similarity to any topic. This allows for an objective measure of how well a summary aligns with a specific topic.
- Control Token Methods for Transformers: To adapt topic control to powerful Transformer architectures, the authors explore three control token strategies:
- Prepending: Adding a special token representing the desired topic at the beginning of the input document.
- Tagging: Identifying and marking the most representative terms for a given topic within the input document using a special
[TAG]token. - Combination: Using both prepending and tagging.
These methods are designed to be easily integrated with existing Transformer models without requiring architectural changes, making them highly adaptable and efficient.
Datasets
- CNN/DailyMail: A standard dataset for abstractive summarization, used here for fine-tuning models.
- Topic-Oriented CNN/DailyMail: A synthetic dataset created by the authors. They combined articles from the Vox dataset (which has topic annotations) to create "super-articles" with summaries that focus on one of two topics. This dataset is crucial for training and evaluating topic-controllable summarization.
- Vox Dataset: Used to extract topic vector representations for the STAS metric.
Key Insights and Ideas
- The Need for Topic-Specific Evaluation: Generic metrics like ROUGE are insufficient for topic-controllable summarization. A dedicated metric like STAS is necessary to accurately assess topical alignment.
- Control Tokens Outperform Embeddings for Transformers: While topic embeddings have been used, they often require significant model modifications. Control tokens, particularly the combined "prepending + tagging" approach, offer a more efficient and effective way to guide Transformer summarization, achieving better results with faster inference.
- Zero-Shot Capability: The control token methods demonstrate remarkable ability to generalize to topics not seen during training, highlighting their robustness and practical utility.
- Efficiency Gains: Control token methods offer an order of magnitude speedup in inference compared to embedding-based approaches, making them more practical for real-world applications.
Key Results
- STAS Reliability: Human evaluation showed a high correlation (Pearson: 0.94, Spearman: 0.87) with the proposed STAS metric, validating its effectiveness.
- Performance Improvement: The proposed BART-based models with control tokens significantly outperformed baseline methods. The
BART_pre+tagmodel achieved the highest STAS score (72.36%) on the topic-oriented dataset. - Inference Speed: Control token methods were nearly 10x faster than embedding-based methods, processing 100 articles in under 40 seconds compared to over 300 seconds.
- Zero-Shot Performance: In a zero-shot setting,
BART_pre+tagachieved a higher ROUGE-1 score (39.22) and STAS score (77.94%) compared to GPT-3.5 (ROUGE-1: 24.43, STAS: 58.16%), demonstrating strong generalization. - Original CNN/DailyMail: Even on the original dataset, the control token methods showed strong performance in terms of STAS, especially in the non-oracle setup.
Limitations and Future Work
- Dataset Creation: The creation of the topic-oriented dataset involved combining articles, which might introduce artifacts. Future work could explore more natural ways to create such datasets.
- Topic Representation: While TF-IDF was used for simplicity, the authors suggest that richer representations like contextual embeddings could further improve STAS.
- Broader Controllable Aspects: Future research could extend these methods to control other aspects of summaries, such as style, entities, or length.
- Arbitrary Topic Tagging: The tagging method currently relies on a pre-defined set of representative terms. Extending this to work with any arbitrary topic without prior knowledge would be a valuable advancement.
Summary updated