TL;DR
This paper introduces the task of generating informative conclusions for argumentative texts, which are often omitted but crucial for quickly understanding arguments in large-scale browsing scenarios. They introduce a new corpus, Webis-ConcluGen-21, and explore both extractive and abstractive methods for generating these conclusions, finding that while abstractive models can be powerful, they require careful encoding of argumentative knowledge.
What Problem Does This Paper Address?
Argumentative texts aim to persuade readers towards a specific conclusion. However, authors often omit these conclusions, expecting readers to infer them. While this might work for individual texts, it hinders efficient comprehension when browsing many texts, such as on search engines or social media. The authors argue that an explicit, informative conclusion acts as a valuable summary of an argumentative text, going beyond just stating a stance to highlighting specific concepts discussed. The challenge is to automatically generate such informative conclusions, especially given the diverse nature of argumentative texts and the implicit ways conclusions are often presented.
Methods
The authors propose two main paradigms for generating informative conclusions:
- Paraphrased Conclusion Generation (Extractive + Paraphrasing): This approach leverages an existing graph-based model (Arg-PageRank) to extract the most central sentence from an argumentative text, hypothesizing it contains the conclusion and key premise. This extracted sentence is then paraphrased using PEGASUS, a model fine-tuned for paraphrasing.
- Abstractive Conclusion Generation: This method involves a three-step process:
- Knowledge Extraction: Identifying key pieces of argumentative knowledge from the text, including the topic, targets (the concepts being argued about), and aspects (subtopics related to the core reasoning).
- Knowledge Encoding: Augmenting the training data with this extracted knowledge using special control codes (e.g.,
<|TOPIC|>,<|TARGETS|>). - Finetuning: Fine-tuning a pre-trained abstractive model (distilled BART) on the augmented corpus. They explore variations encoding different combinations of knowledge (topic, aspects, targets).
Datasets
The paper introduces Webis-ConcluGen-21, a new corpus containing 136,996 pairs of argumentative texts and their conclusions. This corpus is derived from two primary sources:
- Reddit's ChangeMyView (CMV) forum: Original posts serve as argumentative texts, and their titles as conclusions.
- Debate Corpora: Data from platforms like Kialo and Args.me, which facilitate structured debates with explicit conclusions for arguments.
The authors also mention using the XSum corpus for pre-training the distilled BART model.
Key Insights and Ideas
- Informativeness vs. Conciseness Trade-off: The study highlights a tension between generating concise, fluent conclusions and those that are truly informative. Longer, extractive conclusions might capture more nuance but can be less fluent.
- Importance of Argumentative Knowledge: Simply finetuning a general summarization model on argumentative texts is insufficient. Explicitly encoding knowledge about the topic, targets, and aspects of an argument is crucial for generating informative conclusions.
- Limitations of General Summarization Models: Models trained on news summarization (like dbart-XSum) perform poorly on argumentative texts, indicating the need for task-specific training data and approaches.
- Challenges with Implicit Conclusions: Comments on online discussions present a particularly difficult case due to their often implicit nature, reliance on prior context, and mixed stances, requiring more sophisticated modeling.
Key Results
- Automatic Evaluation: The finetuned abstractive models (dbart variants) generally outperform the extractive Arg-PageRank model on metrics like BERTScore and ROUGE, especially when trained on the entire corpus. Knowledge-encoded models show improvements over the base dbart model.
- Manual Evaluation: Expert annotators found that finetuned models generated more valid conclusions than Arg-PageRank. For CMV posts, dbart performed best, while for debates, dbart and dbart-Topic were comparable. Arg-PageRank, however, produced longer conclusions that annotators sometimes preferred for informativeness, especially when background information was scarce.
- Error Analysis: Knowledge-encoded models reduced non-argumentative errors but sometimes increased wrong target errors. Mixed stances in comments led to more stance errors. Arg-PageRank's main error type was non-argumentative sentences.
Limitations and Future Work
- Encoding Knowledge: While control codes help, the authors suggest that more sophisticated encoding strategies are needed to fully leverage argumentative knowledge for generating truly informative conclusions.
- Informativeness vs. Conciseness: The trade-off remains a key challenge, and future work could explore methods to balance these two aspects more effectively.
- Comments: Generating conclusions for comments requires better modeling of argumentative context and explicit stance identification.
- Importance Sampling: The authors suggest future work could benefit from importance sampling of argumentative knowledge, tailored to specific domains or audiences.
Summary updated