On Estimating Variances for Topic Set Size Design

Tetsuya Sakai, Lifeng ShangEVIA@NTCIR 2016 · OpenReview

Introduction

Building Information Retrieval (IR) test collections is a resource-intensive endeavor. Traditionally, researchers have relied on a "rule of thumb" of 50 topics (queries) to ensure a collection's utility. However, this arbitrary threshold does not guarantee statistical power; some tasks may require far fewer topics, while others need significantly more to reliably distinguish between search algorithms.

To optimize this process, researchers use Topic Set Size Design, a framework based on Power Analysis (specifically ANOVA). This allows for the calculation of the exact number of topics needed to detect specific performance differences. The challenge is that these calculations require an estimate of population variance, which typically necessitates a "pilot" dataset. This paper investigates the minimum requirements for such a pilot: how many teams and topics are truly necessary to obtain a reliable variance estimate?

Methodology: Simulating Pilot Studies

The authors conducted a retrospective study using data from the NTCIR-12 Short Text Conversation (STC) task, which included 100 topics and 44 runs from 16 teams. This full dataset served as the "ground truth" for population variance.

To determine the minimum requirements for a pilot study, they employed a "leave-k-out" approach:

  1. Team Reduction: They randomly removed $k$ teams (from 1 to 15) to observe the impact on variance stability.
  2. Topic Reduction: They sampled subsets of topics ($n' = 75, 50, 25, 10$) from the original 100.
  3. ANOVA Estimation: A one-way ANOVA model estimated the within-system variance ($\sigma^2$) for various evaluation measures.
  4. Stability Analysis: 95% confidence intervals (CIs) were calculated over multiple trials to identify when estimates became unstable or biased.

Key Results and Insights

Diversity vs. Volume

A critical finding is that diversity trumps volume. Relying on a single team for a pilot study—even if that team submits multiple runs—is risky and often leads to inaccurate variance estimates. However, the estimates become remarkably robust as soon as runs from even two or three different teams are included.

Measure Sensitivity and Stability

The stability of variance estimates is highly dependent on the specific evaluation metric used. "Navigational" measures like nG@1, which focus only on the top result, are inherently noisier and harder to estimate. In contrast, "informational" measures like Q@10 or nDCG@10, which aggregate data across more ranks, provide much more stable estimates.

Variance stability across different evaluation metrics (a-e) as teams are removed ($k$). Estimates remain remarkably stable until only one or two teams remain ($k=14, 15$), particularly for robust metrics like Q@10.

As shown in the figure above, the variance estimates ($\hat{\sigma}^2$) for measures like Q@10 (e) and nDCG@10 (d) remain nearly flat even as 14 out of 16 teams are removed. For nG@1 (a), the estimate is not only higher but also significantly more volatile as the number of teams decreases.

The "25 Topic" Threshold

For most stable measures, a pilot study with only 25 topics provides a variance estimate nearly as accurate as one with 100 topics. For example, reducing the pilot from 100 to 25 topics for Q@10 resulted in no change to the variance estimate (0.029), while for nDCG@10, it only shifted from 0.034 to 0.035.

Practical Implications

When pilot sizes are extremely small (e.g., 10 topics), the tools tend to overestimate variance. While this leads to recommending more topics than strictly necessary—for instance, suggesting 58 topics for Q@10 when 41 would suffice—it is a "safe" error. It ensures the final experiment will have more than enough statistical power.

In conclusion, researchers can accurately design new IR test collections using modest pilot studies requiring as few as 25 topics and runs from just a few different teams. This approach moves the community away from arbitrary rules of thumb toward evidence-based experimental design.

Summary updated

BibTeX
@inproceedings{sakai2016estimating,
  author = {Tetsuya Sakai and Lifeng Shang},
  title = {{On Estimating Variances for Topic Set Size Design}},
  booktitle = {EVIA@NTCIR},
  year = {2016},
  url = {https://openreview.net/forum?id=Ls6yKXRG9F},
}

Loading related content...

Figure