Abstract
Automated fact-verification systems typically evaluate isolated claims against evidence. However, real-world misinformation often emerges within dialogue, where the context of a question significantly alters the interpretation of a response. We introduce QaDialMoE, a novel framework designed for question-answering dialogue-based fact verification. By leveraging a Mixture of Experts (MoE) architecture and a specialized Attention Guidance module, QaDialMoE effectively models the interactions between questions, responses, and evidence while mitigating common "word-overlap" biases. Our approach achieves state-of-the-art results across multiple benchmarks, including HEALTHVER and FAVIQ.
Introduction
Traditional fact-checking models, such as those trained on the FEVER dataset, often struggle with the nuances of conversational information exchange. These models typically operate on a "Claim vs. Evidence" basis, which fails to account for two critical factors:
- Contextual Neglect: The original question often contains vital clues about which parts of a response are "vulnerable" or require verification.
- The Word-Overlap Trap: Models frequently rely on linguistic shortcuts, predicting a "SUPPORTED" label simply because a response shares many words with the evidence, even if the underlying logic is contradictory.
QaDialMoE addresses these gaps by treating fact verification as a multi-faceted interaction between the user's query, the provided answer, and the retrieved evidence.
Methodology: The QaDialMoE Architecture
The QaDialMoE framework consists of three integrated modules designed to process dialogue context and evidence holistically.
1. Feature Extractor & Prompt Module
The system utilizes a transformer-based encoder (RoBERTa-Large) to generate joint representations of the Response, Question, and Evidence. To handle datasets that lack original questions, a Prompt Module (built on T5) is employed to synthesize relevant queries, ensuring the model can always leverage the question-response-evidence triad.
2. Mixture of Experts (MoE)
Rather than relying on a single monolithic classifier, QaDialMoE employs three specialized "experts":
- Question Expert: Specializes in the interaction between the response and the initial question.
- Evidence Expert: Focuses on the logical consistency between the response and the supporting evidence.
- Global Expert: Synthesizes information from all three elements simultaneously.
3. Management Module with Attention Guidance
A central "manager" assigns weights to these experts to produce the final prediction. To prevent "expert collapse"—where the model over-relies on a single expert—the authors introduce Attention Guidance. This module uses TF-IDF similarity to establish a "prior assumption." For instance, if high word overlap is detected between a response and evidence, the guidance pushes the manager to prioritize the Question Expert to avoid falling into the overlap bias.
Experimental Evaluation
The model was evaluated on three diverse benchmarks: HEALTHVER (real-world COVID-19 Q&A), FAVIQ (ambiguous Wikipedia-based questions), and COLLOQUIAL (conversational adaptations of FEVER claims).
Key Results
QaDialMoE established new state-of-the-art (SOTA) performance across all benchmarks, notably improving accuracy by over 21% on the COLLOQUIAL dataset.
| Dataset | Metric | QaDialMoE | Improvement |
|---|---|---|---|
| HEALTHVER | Accuracy | 84.26% | +3.57% |
| FAVIQ R | Accuracy | 86.0% | +11.4% |
| COLLOQUIAL | Accuracy | 89.5% | +21.8% |
Discussion and Insights
The success of QaDialMoE highlights that the question acts as a filter, focusing the model's attention on the specific entities or claims within a response that require validation.
Furthermore, the Guidance Loss ($L_G$) proved essential. A common problem in MoE is "imbalanced experts," where the manager favors one expert and ignores others (as seen in the figure above). The Guidance Loss forces the manager to distribute attention based on linguistic properties, ensuring all experts contribute meaningfully.
Conclusion
QaDialMoE represents a significant step forward in making fact-verification systems more robust and context-aware. By explicitly modeling dialogue interactions and implementing safeguards against linguistic biases, the framework provides a more reliable tool for combating misinformation in conversational AI. Future work will explore extending this architecture to multi-turn dialogues and more informal social media contexts.
Summary updated