Performance prediction using spatial autocorrelation

Fernando Diaz, Fernando Diaz 0001SIGIR 2007 · DOI

Abstract

Evaluating Information Retrieval (IR) systems traditionally relies on manual relevance judgments, which are costly and difficult to scale. This paper proposes a "zero-judgment" performance predictor based on spatial autocorrelation. By measuring the Moran coefficient ($I_M$) of retrieval scores within a topical document space, the method detects whether a system's rankings align with the Cluster Hypothesis—the principle that topically similar documents should share similar relevance. Experimental results across various TREC collections demonstrate that this approach significantly outperforms existing baselines, particularly in web search scenarios, and remains robust across different languages.

Introduction

The challenge of evaluating search engine performance without human-labeled data is a long-standing problem in Information Retrieval. Traditional metrics like Mean Average Precision (MAP) require exhaustive relevance judgments. As search tasks become more diverse and data volumes explode, automated performance prediction becomes essential. Previous "zero-judgment" predictors often treat document scores as independent variables, ignoring the inherent topical structure of the document collection. This paper addresses this gap by leveraging spatial autocorrelation to evaluate the consistency of retrieval results.

Methodology: The Spatial Autocorrelation Predictor

The core innovation lies in applying the Moran coefficient ($I_M$), a statistical measure from geography, to document retrieval scores. The process involves three main steps:

  1. Spatial Embedding: Documents are mapped into a high-dimensional vector space (e.g., using tf-idf), where proximity represents topical similarity.
  2. Neighborhood Construction: A similarity matrix $W$ is built, typically identifying the 5-nearest neighbors for each document.
  3. Autocorrelation Calculation: The Moran coefficient measures how much a document's score correlates with its neighbors' scores.
Visualizing spatial autocorrelation: (a) shows random distribution (low $I_M$), while (c) shows strong clustering (high $I_M$), indicating a more consistent retrieval.

A high $I_M$ suggests that the retrieval system is assigning similar scores to related documents, adhering to the Cluster Hypothesis. Conversely, a low $I_M$ indicates scattered, potentially noisy results where the system fails to recognize topical clusters.

Experimental Evaluation

The author conducted a large-scale experiment using benchmark collections from the Text REtrieval Conference (TREC), including classic ad hoc tasks (TREC 3-8), web search (Terabyte 2004-2005), and multilingual collections (Spanish and Chinese).

Key Results

  • Web Search Superiority: On the Terabyte 2004 collection, the autocorrelation predictor achieved a Kendall’s $\tau$ of 0.454, vastly outperforming the state-of-the-art "Clarity" baseline (0.139).
  • Cross-Lingual Robustness: The predictor maintained effectiveness in Spanish and Chinese datasets without requiring language-specific tuning.
  • Complementary Utility: When integrated with other prediction paradigms, such as multirun averaging, spatial autocorrelation provided additive gains in prediction accuracy (increasing the adjusted $R^2$).
Comparison of performance prediction paradigms: Global Divergence, Score Perturbation, and Multirun Averaging. Spatial autocorrelation offers a distinct perspective by focusing on local topical consistency.

Discussion and Limitations

While highly effective for general search, the method encountered difficulties in specialized tasks like Entity Retrieval (Enterprise Expert Search). This suggests that simple co-occurrence-based document graphs may not capture the complex relationships between entities. Additionally, the predictor assumes that relevance is clustered; in tasks requiring high diversity, a high $I_M$ might not correlate with user satisfaction. Future work may explore how different neighborhood sizes ($k$) or similarity measures affect the predictor's sensitivity.

Conclusion

Spatial autocorrelation provides a robust, system-agnostic framework for predicting IR performance. By quantifying the Cluster Hypothesis, it offers a powerful tool for automated system monitoring and optimization, especially in large-scale web environments where manual labeling is infeasible.

Summary updated

BibTeX
@inproceedings{diaz2007performance,
  author = {Fernando Diaz and Fernando Diaz},
  title = {{Performance prediction using spatial autocorrelation}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2007},
  doi = {10.1145/1277741.1277841},
}
Figure