Optimizing Question Answering Accuracy by Maximizing Log-Likelihood

Matthias H. Heie, Edward W. D. Whittaker, Sadaoki FuruiACL 2010

Introduction

In factoid Question Answering (QA), identifying the "question type" is a critical step for filtering candidate answers. Historically, researchers have relied on hand-crafted rules or supervised learning on manually labeled datasets. However, manual rules are difficult to scale across languages, and supervised learning requires expensive human annotation. This paper proposes an unsupervised alternative: a greedy clustering algorithm that automatically categorizes question-answer (q-a) pairs by maximizing the log-likelihood of a statistical model.

Methodology

The authors utilize a statistical QA framework where the probability of an answer $A$ given a question $Q$ is modeled based on question-type features ($W$) and information-bearing features ($X$). The core of the approach is the Filter Model, $P(W|A)$. To make this model tractable, q-a pairs are grouped into clusters ($C_E$).

The proposed greedy clustering algorithm works by iteratively moving q-a pairs into different clusters (including potential new clusters) and calculating the log-likelihood ($LL$) on a development set. If a move increases the $LL$, it is kept. This process continues until no further improvements can be made, effectively discovering the optimal number and composition of question categories without human intervention.

Key Insights: The LL-MRR Correlation

A fundamental finding of this research is the extremely strong correlation between the model's log-likelihood and actual QA accuracy, measured by Mean Reciprocal Rank (MRR).

Figure 1: Scatter plot showing the strong correlation (ρ = 0.86) between Mean Reciprocal Rank (MRR) and Log-Likelihood (LL).

As shown in Figure 1, the correlation coefficient of 0.86 justifies using $LL$ as a proxy for accuracy during training. This relationship allows the unsupervised algorithm to optimize for a statistical metric that directly translates to better real-world performance.

Experimental Results

The model was trained on 2,016 q-a pairs and evaluated using TREC QA data. The automatic clustering method was compared against several baselines, including manual rule-based clustering and a "one-cluster-per-pair" approach.

Table 1: Comparison of different clustering configurations, showing that automatic clustering achieves the highest MRR with a small number of clusters.

As summarized in Table 1, the automatic configuration achieved an MRR of 0.281, representing a 7% relative improvement over the manual baseline (0.262). Notably, the algorithm achieved this with only 4 clusters, demonstrating that data-driven categories can be more efficient and accurate than human intuition. For example, the model occasionally grouped "who" and "where" questions together when the answers (such as a country name for a sports team) justified the overlap.

Conclusion

This study proves that an unsupervised, data-driven approach can outperform traditional manual categorization in factoid QA. By maximizing log-likelihood, the system discovers optimal question-type structures that enhance ranking accuracy. Future work will focus on expanding the model to handle multi-word answers and more complex question types like "how" and "why."

Summary updated

BibTeX
@inproceedings{heie2010optimizing,
  author = {Matthias H. Heie and Edward W. D. Whittaker and Sadaoki Furui},
  title = {{Optimizing Question Answering Accuracy by Maximizing Log-Likelihood}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2010},
}
Figure