Abstract
Information filtering systems based on statistical retrieval models usually compute a numeric score indicating how well each document matches each profile. Documents with scores above profile-specific dissemination thresholds are delivered. An optimal dissemination threshold is one that maximizes a given utility function based on the distributions of the scores of relevant and non-relevant documents. The parameters of the distribution can be estimated using relevance information, but relevance information obtained while filtering is biased. This paper presents a new method of adjusting dissemination thresholds that explicitly models and compensates for this bias. The new algorithm, which is based on the Maximum Likelihood principle, jointly estimates the parameters of the density distributions for relevant and non-relevant documents and the ratio of the relevant document in the corpus. Experiments with TREC-8 and TREC-9 Filtering Track data demonstrate the effectiveness of the algorithm.
Introduction: The Threshold Problem
In information filtering—such as personalized news feeds or automated legal discovery—systems must decide in real-time whether an incoming document is relevant enough to show a user. This decision typically relies on a dissemination threshold. If a document's similarity score exceeds this threshold, it is delivered; otherwise, it is discarded. Setting an optimal threshold is critical: too high, and the user misses valuable information; too low, and the user is overwhelmed by noise.
The Core Challenge: Selection Bias
A fundamental difficulty in adaptive filtering is that systems only receive relevance feedback for documents they actually deliver. This creates a significant selection bias (or survivorship bias). If a system's threshold is high, it only observes the "cream of the crop."
As shown in Figure 3, calculating parameters like the mean score from only delivered documents leads to an artificially high estimate. This missing data—the relevant documents that fell just below the current threshold—causes the system to become overly conservative, potentially entering a "vicious cycle" where thresholds remain too high because the system never sees the data needed to justify lowering them.
Methodology: Modeling Distributions with MLE
To address this bias, the authors (Zhang and Callan, SIGIR 2001) proposed a model based on two key distributional assumptions:
- Relevant document scores follow a Gaussian (Normal) distribution.
- Non-relevant document scores follow an Exponential distribution.
The core innovation is the use of Maximum Likelihood Estimation (MLE). Instead of simple parameter calculation, the model treats each delivered document as a data point consisting of its score, the user's relevance judgment, and the specific threshold in place at the time of delivery. The MLE approach finds the parameters (mean, variance, and the ratio of relevant documents) that make this specific, biased set of observed data most likely to have occurred.
Because the resulting likelihood equations lack a closed-form solution, the authors employ Conjugate Gradient Descent, a numerical optimization technique, to find the optimal parameters.
Experimental Evaluation
The algorithm was tested using standard benchmarks from the Text REtrieval Conference (TREC):
- OHSUMED (TREC-9): ~350,000 medical articles.
- Financial Times (TREC-8): ~210,000 articles.
Key Results
The MLE approach demonstrated substantial improvements over heuristic baselines:
- Utility Gains: On the OHSUMED dataset, the MLE method achieved an average utility (T9U' metric) of 8.17, more than double the 3.25 achieved by baseline methods.
- Improved Recall: By accurately modeling the bias, the system was much more effective at "recovering" and lowering thresholds that were set too high, leading to significantly higher recall (0.137 vs 0.080) without a proportional loss in precision.
- Computational Efficiency: The algorithm is efficient enough for real-time use, processing four years of data for dozens of profiles in approximately 21 minutes on hardware of the era (500 MHz Pentium III).
Conclusion
The "Maximum Likelihood Estimation for Filtering Thresholds" paper provides a robust mathematical framework for overcoming the inherent bias in relevance feedback. By explicitly modeling the sampling process, filtering systems can learn more accurate score distributions, leading to more effective and adaptive information delivery. While it relies on specific distributional assumptions and requires a "minimum delivery" heuristic to avoid cold-start issues, it remains a foundational approach in the field of information retrieval.
Summary updated