TL;DR
This paper tackles the challenge of ranking text passages not just by relevance, but by how convincing their arguments are, especially for topics with multiple viewpoints. They introduce a new, large-scale dataset and demonstrate that training models directly on pairwise comparisons of convincingness outperforms regression-based approaches, and that filtering noisy crowd-sourced annotations for transitivity is more effective than confidence-based filtering.
What Problem Does This Paper Address?
In online search, results are typically ranked by relevance. However, for complex or polarizing topics, users might benefit from seeing arguments that are not only relevant but also well-reasoned and persuasive, representing different sides of an issue. This is particularly relevant for "multi-perspective search" features. The challenge lies in automatically identifying and ranking passages based on their argumentative convincingness, a task that goes beyond simple relevance or stance detection. Existing datasets for this task are relatively small and limited in topical coverage, making it difficult to train robust models. Furthermore, crowd-sourced annotations, while valuable, can be noisy, requiring effective methods for data cleaning and model training.
Methods
The authors build upon a "sum-of-word-embeddings" approach for representing passages, extending it with a Feed Forward Neural Network (FFNN) to predict a convincingness score. For training, they explore two main strategies using pairwise annotated data:
- Direct Pairwise Training: A classification objective using cross-entropy, where the model learns to predict which passage in a pair is more convincing.
- Regression-Based Training: This involves first generating real-valued scores for each passage and then training a regression model. They investigate two methods for generating these scores:
- PageRank (PR): Based on directed graphs derived from pairwise annotations.
- Win-Rate (WR): A simpler score calculated by how often a passage is rated more convincing than its competitors.
A significant part of their work involves creating and annotating a new dataset and then evaluating different data filtering strategies (e.g., removing cycles in annotation graphs, filtering based on annotation confidence) and their impact on model performance.
Datasets
- UKP Dataset: Used for initial evaluation and comparison with prior state-of-the-art models. This dataset contains passages for 16 topics.
- Newly Introduced Dataset: This is the primary dataset used for experiments. It's substantially larger, covering 3,234 topics with approximately 30,000 passages. It was constructed from data collected for Bing's multi-perspective search feature and annotated for stance and convincingness.
Key Insights and Ideas
- Pairwise is Powerful: Directly training a model on pairwise comparisons of convincingness (using a classification objective) yields better results than trying to regress to a single score for each passage. This suggests that the relative judgment is more informative than an absolute score.
- Win-Rate as a Regression Target: If regression is the chosen path, a simple "Win-Rate" score is a more effective target than the previously proposed PageRank-based scores, and can even be competitive with direct pairwise training.
- Filtering for Transitivity is Key: When dealing with noisy crowd-sourced annotations, filtering out passage sets that exhibit cycles (i.e., A > B, B > C, C > A) in their pairwise comparisons is more effective for improving model performance than filtering based on individual annotation confidence scores. This highlights the importance of structural consistency in the annotations.
- Convincingness is Distinct from Relevancy: The paper suggests that argument convincingness is a separate attribute from topical relevancy. While relevancy helps select relevant passages, a dedicated convincingness model is needed to effectively rank arguments on different sides of an issue.
Key Results
- The proposed model (Sum-of-Embeddings + FFNN) achieved state-of-the-art results on the UKP dataset for convincingness ranking.
- On their newly created, larger dataset, direct pairwise training with a cross-entropy objective outperformed regression-based methods.
- Regression to Win-Rate was superior to regression to PageRank.
- Filtering out passage sets with cycles in their directed graphs significantly improved performance.
- Filtering based on individual annotation confidence scores could actually hurt performance, but weighting training examples by confidence was beneficial.
Limitations and Future Work
- Dataset Size and Noise: While their new dataset is large, it still relies on crowd-sourced annotations, which inherently contain noise. The paper explores filtering strategies to mitigate this but acknowledges the trade-off between noise reduction and data availability.
- Combining Convincingness and Relevancy: The paper notes that convincingness and relevancy are distinct attributes. Future work could explore how to best combine these for an optimal user experience in search.
- Model Robustness: The authors suggest further investigation into the best practices for combining different attributes (like convincingness and relevancy) for search engine design.
Summary updated