Abstract
Evaluating Information Retrieval (IR) systems traditionally relies on manual relevance judgments, which are costly and difficult to scale. This paper proposes a "zero-judgment" performance predictor based on spatial autocorrelation. By measuring the Moran coefficient ($I_M$) of retrieval scores within a topical document space, the method detects whether a system's rankings align with the Cluster Hypothesis—the principle that topically similar documents should share similar relevance. Experimental results across various TREC collections demonstrate that this approach significantly outperforms existing baselines, particularly in web search scenarios, and remains robust across different languages.
Introduction
The challenge of evaluating search engine performance without human-labeled data is a long-standing problem in Information Retrieval. Traditional metrics like Mean Average Precision (MAP) require exhaustive relevance judgments. As search tasks become more diverse and data volumes explode, automated performance prediction becomes essential. Previous "zero-judgment" predictors often treat document scores as independent variables, ignoring the inherent topical structure of the document collection. This paper addresses this gap by leveraging spatial autocorrelation to evaluate the consistency of retrieval results.
Methodology: The Spatial Autocorrelation Predictor
The core innovation lies in applying the Moran coefficient ($I_M$), a statistical measure from geography, to document retrieval scores. The process involves three main steps:
- Spatial Embedding: Documents are mapped into a high-dimensional vector space (e.g., using tf-idf), where proximity represents topical similarity.
- Neighborhood Construction: A similarity matrix $W$ is built, typically identifying the 5-nearest neighbors for each document.
- Autocorrelation Calculation: The Moran coefficient measures how much a document's score correlates with its neighbors' scores.
A high $I_M$ suggests that the retrieval system is assigning similar scores to related documents, adhering to the Cluster Hypothesis. Conversely, a low $I_M$ indicates scattered, potentially noisy results where the system fails to recognize topical clusters.
Experimental Evaluation
The author conducted a large-scale experiment using benchmark collections from the Text REtrieval Conference (TREC), including classic ad hoc tasks (TREC 3-8), web search (Terabyte 2004-2005), and multilingual collections (Spanish and Chinese).
Key Results
- Web Search Superiority: On the Terabyte 2004 collection, the autocorrelation predictor achieved a Kendall’s $\tau$ of 0.454, vastly outperforming the state-of-the-art "Clarity" baseline (0.139).
- Cross-Lingual Robustness: The predictor maintained effectiveness in Spanish and Chinese datasets without requiring language-specific tuning.
- Complementary Utility: When integrated with other prediction paradigms, such as multirun averaging, spatial autocorrelation provided additive gains in prediction accuracy (increasing the adjusted $R^2$).
Discussion and Limitations
While highly effective for general search, the method encountered difficulties in specialized tasks like Entity Retrieval (Enterprise Expert Search). This suggests that simple co-occurrence-based document graphs may not capture the complex relationships between entities. Additionally, the predictor assumes that relevance is clustered; in tasks requiring high diversity, a high $I_M$ might not correlate with user satisfaction. Future work may explore how different neighborhood sizes ($k$) or similarity measures affect the predictor's sensitivity.
Conclusion
Spatial autocorrelation provides a robust, system-agnostic framework for predicting IR performance. By quantifying the Cluster Hypothesis, it offers a powerful tool for automated system monitoring and optimization, especially in large-scale web environments where manual labeling is infeasible.
Summary updated