Top-Rank-Focused Adaptive Vote Collection for the Evaluation of Domain-Specific Semantic Models

Pierangelo Lombardo, Alessio Boiardi, Luca Colombo, Angelo Schiavone, Nicolò TamagnoneEMNLP 2020 · DOI

Introduction

Evaluating domain-specific semantic models, such as those used in recruitment or legal tech, is notoriously resource-intensive. Standard benchmarks often fail to capture the nuances of niche fields like biomedical research or oil and gas. Furthermore, real-world applications like search engines prioritize the accuracy of top-ranked results over the entire list. This paper addresses these challenges by introducing an adaptive "tournament-style" protocol for human vote collection that focuses effort on the most relevant items while introducing weighted metrics to penalize top-rank errors more heavily.

Methodology

The proposed framework consists of three main components: adaptive vote collection, weighted ranking metrics, and a stochastic transitivity model for simulation.

Overview of the proposed framework for adaptive vote collection and model evaluation.

Adaptive Vote Collection

Instead of uniform random sampling, the protocol uses a multi-ballot system. In the initial round, pairs are sampled uniformly. In subsequent rounds, only the "winners"—items deemed more related by annotators—advance. This concentrates human effort on distinguishing between highly relevant items, effectively increasing the resolution where it matters most: the top ranks.

Weighted Ranking Metrics

Standard metrics like Spearman’s $\rho$ treat all rank positions equally. The authors extend these by incorporating a weight function $f(n; n_0) = 1/(n + n_0)^2$. This ensures that a mismatch at rank 1 is significantly more costly than a mismatch at rank 100, aligning the evaluation with downstream recommendation tasks.

Simulation and Datasets

To validate the approach without immediate large-scale human trials, the authors developed a stochastic transitivity model to simulate "noisy" human voters. This model accounts for subjectivity, oversight, and the difficulty of comparing highly similar items.

Underlying similarity distributions used to simulate human voting behavior.

The methodology was tested using tokens from the recruitment domain (INDA) and synthetic votes generated based on various underlying distributions, including Exponential and Power Law models.

Key Results

The adaptive approach significantly outperforms uniform sampling in top-rank accuracy. For an exponential similarity distribution, the weighted Spearman’s $\rho_w$ improved from 0.778 (Uniform) to 0.945 (Adaptive). Crucially, this gain in top-rank precision does not come at the expense of overall rank stability; the unweighted $\rho$ remains stable, indicating that the protocol efficiently redistributes resources without breaking the model's general understanding of the dataset.

Conclusion

Adaptive vote collection offers a path toward building high-quality, domain-specific evaluation benchmarks with fewer resources. By focusing on the "survival of the fittest" items and employing weighted metrics, researchers can better align model evaluation with real-world performance requirements. Future work will focus on large-scale human validation and refining parameter tuning for different domain sizes.

Summary updated

BibTeX
@inproceedings{lombardo2020top,
  author = {Pierangelo Lombardo and Alessio Boiardi and Luca Colombo and Angelo Schiavone and Nicolò Tamagnone},
  title = {{Top-Rank-Focused Adaptive Vote Collection for the Evaluation of Domain-Specific Semantic Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.249},
}
Figure