TL;DR
This paper introduces the Tree Aggregation Transformer, a novel recursive, tree-structured self-attention model for Answer Sentence Selection (AS2). By explicitly incorporating syntactic parse tree structures, the model achieves state-of-the-art results on formal QA datasets without relying on transfer learning, demonstrating the value of syntactic information for this task.
What Problem Does This Paper Address?
Answer Sentence Selection (AS2) is a crucial task in question answering, aiming to identify which sentence in a collection of candidate answers actually contains the answer to a given question. While recent top-performing models leverage powerful transformer architectures and transfer learning, they largely ignore the inherent syntactic structure of language. This paper hypothesizes that incorporating syntactic structure, specifically through parse trees, can improve AS2 performance, especially in tasks where language is more formal and structured.
Methods
The core innovation is the Tree Aggregation Transformer. This model augments standard token embeddings with syntactic information, including part-of-speech tags, syntactic categories, and tree levels. This enriched representation is then processed by a recursive self-attention layer. Unlike previous tree-structured models that might use separate parameters for each level or non-recursive approaches, this model uses a single set of self-attention parameters recursively across the entire parse tree. This allows it to efficiently capture hierarchical syntactic information. The model then aggregates these representations bottom-up to form a final sentence embedding.
Datasets
The paper evaluates its model on six English-language benchmark datasets:
- TrecQA (Clean) and WikiQA (Clean): Widely used datasets for Question Answering (QA).
- YahooCQA, SemEval 2015 CQA, SemEval 2016 CQA, and SemEval 2017 CQA: Datasets from the Community Question Answering (CQA) domain, characterized by more informal and potentially noisy user-generated text.
Key Insights and Ideas
- Syntactic structure matters for AS2: The model's success on formal QA datasets (TrecQA, WikiQA) strongly suggests that explicit syntactic information is beneficial for understanding sentence relationships in AS2.
- Recursion is key for efficiency: Using a single set of recursive self-attention parameters across tree levels is more efficient than training separate parameters for each level, as seen in prior work.
- Formal vs. Informal Text: The model performs exceptionally well on formal QA datasets but shows limitations on CQA datasets. This highlights a crucial challenge: adapting tree-structured models to the noise, informality, and longer text lengths often found in user-generated content.
- Probing reveals what's learned: The probing experiments demonstrate a strong correlation between the model's ability to absorb syntactic information and its performance gains in AS2, confirming the effectiveness of the tree-structured approach when the input is amenable.
Key Results
- State-of-the-art on QA: The Tree Aggregation Transformer achieves new state-of-the-art results on the clean versions of TrecQA and WikiQA, outperforming strong baselines without relying on transfer learning.
- Competitive on CQA: The model matches or exceeds state-of-the-art performance on 2 out of 4 CQA datasets, demonstrating its broad applicability.
- Ablation Study: Removing syntactic embeddings significantly degrades performance, underscoring their importance.
- Probing Task Performance: Tree-structured representations achieved significantly higher F1 scores than sequential representations on syntactic probing tasks for datasets where the model excelled in AS2.
Limitations and Future Work
- CQA Dataset Challenges: The model's performance on CQA datasets is limited by the informal language, potential spelling errors, and longer text lengths, which may hinder its ability to leverage tree structures effectively.
- Parser Generalization: The performance on some datasets might be affected by the generalization capabilities of the pre-trained parsers used to generate the trees.
- Future Work: The authors suggest further research is needed to adapt tree-structured representations to noisy user-generated text. They also note that while their model's recursion makes it aware of input length, this information doesn't necessarily translate to improved AS2 performance.
Summary updated