Abstract
Training question-answering (QA) models under weak supervision often encounters the "spurious solution" problem, where incorrect reasoning paths accidentally lead to the correct answer. This paper proposes a novel framework that leverages mutual information maximization to filter out these misleading signals. By introducing a "question reconstructor" that evaluates whether a candidate solution contains sufficient semantic information to recover the original question, the model effectively identifies logically grounded solutions. Experimental results across reading comprehension, discrete reasoning, and semantic parsing benchmarks demonstrate significant improvements in accuracy and robustness compared to traditional weak supervision methods.
Introduction: The Spurious Solution Problem
In many practical QA applications, full supervision—where the exact derivation path (solution) for an answer is known—is expensive or impossible to obtain. Instead, researchers rely on weak supervision, where only the question and the final answer are provided. However, this often leads to "spurious solutions": incorrect logic that happens to produce the right answer. For instance, in a math-based question, a model might subtract two unrelated numbers to get a result that matches the ground truth. Training on such "lucky guesses" degrades the model's ability to generalize to new contexts where these coincidences do not occur.
Methodology: Mutual Information Maximization
The core contribution of this work is a framework based on Mutual Information (MI) Maximization. The intuition is that a valid solution should be semantically linked to the question; therefore, it should be possible to reconstruct the question from the solution and the context. If a solution is spurious, it will lack the necessary semantic concepts to rebuild the original query.
The Training Cycle
The architecture consists of two main components: a Task-specific Model ($\theta$) and a Question Reconstructor ($\phi$). The training follows a two-step cycle:
- Reconstructor Training: The Task-specific model proposes candidate solutions. The reconstructor (based on BART-base) is trained to maximize the probability of generating the original question given these solutions and the context.
- QA Model Training: The Task-specific model is then updated by prioritizing solutions that the reconstructor finds most "believable" (i.e., those that yield the highest likelihood of the original question).
Unified Solution Encoding
To handle diverse tasks like SQL generation or multi-step arithmetic, the authors implemented a unified solution encoding scheme. This includes a custom attention mask and placeholder tokens (<span>) to ensure the model learns the relationship between the solution's logic and the context's semantics without being misled by identical-looking text strings.
Experimental Setup and Results
The method was evaluated on four diverse benchmarks to prove its versatility:
- Quasar-T & WebQuestions: Multi-mention reading comprehension.
- DROP: Discrete reasoning (math, counting, sorting) over paragraphs.
- WikiSQL: Semantic parsing of natural language into SQL queries.
Key Findings
- Performance Gains: The MI-based approach consistently outperformed standard baselines like Hard EM and Maximum Marginal Likelihood (MML). On WikiSQL, the model achieved 85.6% execution accuracy, nearly matching the 86.2% achieved by fully supervised models.
- Robustness to Noise: On a "hard" version of WikiSQL where questions had an average of over 1,500 possible solutions, baseline accuracy plummeted to 14%, while the MI-based approach maintained high performance.
- Logic Verification: Human evaluation on the DROP dataset confirmed that the model produced the correct logic for 92% of its correct answers, a significant improvement over heuristics-based methods.
Discussion and Conclusion
By framing the spurious solution problem as a mutual information maximization task, this work provides a robust way to train QA models under weak supervision. The reconstruction-based filtering provides a more stable training signal than reinforcement learning-based approaches. While the method is currently limited by its dependency on a precomputed set of candidate solutions and increased computational overhead during training, it represents a significant step toward semantically grounded weak supervision.
Summary updated