TL;DR
This paper introduces a novel task: automatically generating section titles for Wikipedia articles. They propose and evaluate several encoder-decoder models, finding that RNN decoders excel at abstractive titles and transformer decoders at extractive ones. Crucially, they also show that these models can generate useful section embeddings for clustering, with a new state-constraint loss function improving embedding quality.
What Problem Does This Paper Address?
Wikipedia, a vast repository of information, often suffers from articles with poor or inconsistent section headings, hindering organization and discoverability. While automated methods exist for improving Wikipedia content, they often focus on adding new content rather than organizing existing information. This paper tackles this by proposing a new task: generating section titles for Wikipedia articles. This task is not only useful for improving Wikipedia itself but also for organizing other unstructured text collections. Furthermore, the paper explores how the internal representations (embeddings) learned by these generative models can be used for clustering sections with similar content, offering a flexible way to create paragraph-level embeddings.
Methods
The core of the paper's methodology lies in using encoder-decoder architectures initialized with a pre-trained BERT encoder. They experiment with four decoder variants:
- TRANS: A standard transformer decoder, similar to BERT2RND, which allows the decoder to attend over the entire encoder output at each step. This is their primary baseline.
- RNN: An RNN (specifically a GRU) decoder. Unlike the transformer, it only receives the initial CLS token from the encoder, forcing it to encode all information into a single state. This model does not use attention over the input.
- RNN+SC (State Constraint): This model builds upon the RNN decoder but introduces a novel loss function. This "state constraint" loss encourages the initial and final hidden states of the RNN decoder to be similar. The intuition is that this keeps the decoder "on topic" and forces it to encode meaningful information into a single state, improving semantic encoding.
- RNN+ATTN: This model combines a transformer encoder with an RNN decoder, but crucially, it does incorporate an attention mechanism over the encoder's output.
The paper also introduces a metric called extractive score to quantify how much of a generated title can be directly copied from the input section text.
Datasets
The primary dataset used is a collection of articles from English Language Wikipedia, collected on June 25, 2019. They filtered out articles with "redirect" and sections with titles shorter than 2 characters. The data was split into train (80%), test (10%), and development (10%) sets, resulting in approximately 11.43 million articles for training.
For comparison and to evaluate their models on a more established task, they also use the Gigaword Corpus. This corpus is typically used for abstractive summarization, where the task is to generate an article title given the first sentence of a newspaper article.
Key Insights and Ideas
- RNN vs. Transformer Decoders: The paper reveals a trade-off: RNN decoders (without attention) are better at generating short, abstractive titles, while transformer decoders (with attention) excel at longer, extractive titles.
- The Power of State Constraint: The novel RNN+SC model, with its state constraint loss, significantly improves the quality of generated section embeddings, making them more suitable for clustering. This suggests that forcing the model to encode information into a single, consistent state is beneficial for semantic representation.
- Attention Facilitates Copying, Not Necessarily Semantics: The analysis of extractive scores suggests that attention mechanisms in models like TRANS and RNN+ATTN primarily help by enabling the model to copy text from the input. This is great for extractive titles but might not be as crucial for generating truly abstractive or semantically rich embeddings.
- Generic Headings Make Wikipedia Abstractive: The prevalence of common headings like "History" in Wikipedia makes the section title generation task more abstractive compared to datasets like Gigaword, which tend to have more extractive titles.
- Embeddings for Clustering: The paper demonstrates that embeddings extracted from the internal states of these generative models can be effectively used for clustering sections, outperforming off-the-shelf methods like Doc2Vec and TF-IDF.
Key Results
- Section Heading Generation: The RNN+ATTN model performed best overall on standard summarization metrics (Rouge-1, Rouge-L, exact match) for section title generation. The TRANS and RNN+SC models performed similarly and better than the basic RNN.
- Length Matters: For short headings (1-5 tokens), RNN decoders (especially RNN+SC) outperformed transformer decoders. For mid-range headings (5-10 tokens), attention-based decoders (TRANS, RNN+ATTN) showed an advantage. For longer headings (>10 tokens), Rouge-L scores declined across all models.
- Clustering Performance: The RNN+SC model achieved the best results on Wikipedia section clustering metrics (Homogeneity, Completeness, V-measure, ARI), outperforming other models and off-the-shelf embedding methods.
- Gigaword Comparison: On the Gigaword dataset, the TRANS model performed best overall, and attention-based models showed higher partial correlations between Rouge-L and extractive score, reinforcing the idea that attention aids extraction.
Limitations and Future Work
- Focus on English Wikipedia: The current work is limited to English Wikipedia. The authors suggest exploring other language editions, which might have different characteristics and benefit from similar tools.
- Ignoring Hierarchical Structure: The paper discards the hierarchical nature of Wikipedia sections. Incorporating this could potentially improve model performance, though it might reduce generalizability.
- Clustering as Primary Evaluation: While clustering is a strong evaluation for embeddings, the authors suggest further work is needed to assess their usefulness on other tasks like retrieval, similarity measurement, or text classification.
- Extractive vs. Abstractive Trade-off: The paper highlights the tension between models that excel at copying (extractive) and those that can generate novel text (abstractive). Future work could explore how to better balance these for tasks requiring both.
Summary updated