TL;DR
This paper tackles the challenge of coreference resolution in conversational machine reading comprehension (CMRC) by proposing novel graph-based methods that integrate coreference information into pre-trained language models (PLMs). Their "evidence-enhanced" and "fusion" models significantly improve CMRC performance by explicitly modeling referential dependencies, offering a more interpretable and robust approach than standard end-to-end PLMs.
What Problem Does This Paper Address?
Conversational Machine Reading Comprehension (CMRC) requires models to not only understand a given text but also to follow the flow of a conversation, which often involves understanding pronoun references (anaphora). For example, to answer "Did she have any visitors?", a model needs to know who "she" refers to, often based on previous turns in the conversation or the article context. Existing PLMs, while powerful, struggle with this "logical comprehension" and coreference resolution, often relying on end-to-end methods that lack interpretability and can fail when referential entities are distant or ambiguous. This paper aims to bridge this gap by developing methods that explicitly incorporate coreference information to enhance CMRC models.
Methods
The authors propose a novel coreference graph approach. This graph represents coreferential entities within the conversation history and article context separately.
- Coreference Graph Construction: They extract reference words and organize them into graph structures. For article context, entities are clustered, and the first encountered antecedent acts as the head node. For conversation history, entities are connected sequentially based on the turn order.
- Evidence-Enhanced Method: This textual-level approach enriches the PLM's input. It involves two strategies:
- Evidence Generator (EG): Creates new text to elaborate on pronouns (e.g., "She is Jessica"). This generated text is then added to the PLM's input.
- Enhanced Question (EQ): Rewrites the question by replacing pronouns with their resolved referential entities.
- Fusion Model: This structural-level method combines the PLM with a Graph Neural Network (GNN). The PLM acts as an encoder to extract sequence features, while the GNN processes the coreference graph to extract structural features. These features are then fused using learnable weights to enhance the PLM's reasoning capabilities.
Datasets
The experiments were conducted on the CoQA dataset. Two specific subsets were used:
- CoQA all: The complete dataset, used to evaluate the fusion model's overall performance, including on questions without pronouns.
- CoQA pronoun-containing (38% of CoQA all): A newly curated subset specifically designed to evaluate the models' coreference resolution capabilities for anaphora.
Key Insights and Ideas
- Explicit Coreference Modeling is Crucial: The paper strongly suggests that simply relying on the attention mechanisms of standard PLMs is insufficient for robust coreference resolution in CMRC. Explicitly modeling these relationships via graphs is key.
- Graph Structure Enhances Reasoning: Integrating coreference graphs, whether by generating textual evidence or fusing graph embeddings, significantly boosts performance, demonstrating the value of structured knowledge.
- Fusion Model Offers Robustness: The fusion model, which combines PLM and GNN embeddings, not only improves coreference resolution but also maintains strong performance on general CMRC tasks, indicating its robustness and ability to handle both coreference-heavy and standard questions.
- Interpretability Gains: The proposed methods, particularly the evidence-enhanced approach, offer a more interpretable way to answer questions by providing explicit reasoning steps, addressing a known limitation of end-to-end PLMs.
- GCN Outperforms GAT: For graph embedding, the Graph Convolutional Network (GCN) generally outperformed the Graph Attention Network (GAT) on the specific graph structures used, suggesting that simpler, convolution-based approaches might be more effective for these structured graphs.
Key Results
- The proposed graph-combined methods significantly outperform baseline PLM methods (BERT and RoBERTa).
- On the CoQA pronoun-containing dataset, the methods achieved improvements of up to 2.6 F1 score on BERT and 0.7 F1 score on RoBERTa compared to their respective baselines.
- The fusion model achieved the best performance among the proposed methods and showed improvement even on the full CoQA dataset.
- Qualitative analysis (Figure 4, Case Study) shows that the fusion model can accurately predict answer spans, whereas baselines without coreference resolution make incorrect predictions.
Limitations and Future Work
- Limitations in Fully Correct Predictions: While the models show significant improvement, the authors note that there are still limitations in making fully correct answer predictions, especially when the F1 threshold for correctness is set to 1.
- PLM Architecture Dependence: The improvement for RoBERTa with the evidence-enhanced method was noted as "limited," suggesting that the effectiveness of certain methods might depend on the specific PLM architecture.
- Future Work: The paper implicitly suggests further exploration into optimizing the fusion of graph and textual information, potentially investigating more advanced GNN architectures or different fusion strategies. The focus on interpretability also opens avenues for developing more transparent and explainable AI systems for CMRC.
Summary updated