Collaborative Ranking: A Case Study on Entity Linking

Zheng Chen, Heng JiEMNLP 2011 · OpenReview

Abstract

In this paper, we present a new ranking scheme, collaborative ranking (CR). In contrast to traditional non-collaborative ranking scheme which solely relies on the strengths of isolated queries and one stand-alone ranking algorithm, the new scheme integrates the strengths from multiple collaborators of a query and the strengths from multiple ranking algorithms. We elaborate three specific forms of collaborative ranking, namely, micro collaborative ranking (MiCR), macro collaborative ranking (MaCR) and micro-macro collaborative ranking (MiMaCR). Experiments on entity linking task show that our proposed scheme is indeed effective and promising.

Introduction

Traditional ranking systems often treat individual queries as isolated events. In tasks such as Entity Linking (EL)—where a mention like "Michael Jordan" must be mapped to a specific entry in a Knowledge Base (KB)—this isolation leads to two primary issues: information scarcity and model bias. A single document may lack sufficient context to disambiguate an entity, and no single ranking algorithm (pointwise, pairwise, or listwise) is universally optimal.

The general framework for Entity Linking, highlighting the candidate ranking stage.

To address these limitations, we propose Collaborative Ranking (CR). This framework leverages "collaborators" across two dimensions: the micro-level (similar queries or documents) and the macro-level (diverse ranking algorithms). By aggregating information from these sources, the system can make more robust ranking decisions.

Methodology: Three Forms of Collaborative Ranking

We define three distinct implementations of the CR framework:

  • Micro Collaborative Ranking (MiCR): Focuses on query-level collaboration. For a target query, the system identifies "collaborators" (other documents in a large corpus mentioning the same name string). Using graph-based or agglomerative clustering, these documents are grouped. The final ranking score for a candidate object is an aggregate (e.g., max, min, or average) of scores from the query and its collaborators.
  • Macro Collaborative Ranking (MaCR): Focuses on ranker-level collaboration. This is an ensemble approach that integrates multiple ranking functions (e.g., TF-IDF and supervised ListNet) using composite functions like voting or weighted averaging.
  • Micro-Macro Collaborative Ranking (MiMaCR): A hybrid approach that first computes micro-ranking scores for each individual algorithm across query collaborators, then performs a macro-integration across the different algorithms.
Comparison of ranking schemes: (a) non-collaborative, (b) micro, (c) macro, and (d) micro-macro collaborative ranking.

Experimental Setup

The proposed schemes were evaluated on the Entity Linking task using the TAC-KBP (Text Analysis Conference - Knowledge Base Population) tracks:

  • Data: TAC-KBP 2009 (3,904 queries) for training/dev and TAC-KBP 2010 (2,250 queries) for testing.
  • Knowledge Base: A snapshot of English Wikipedia (Oct 2008) containing approximately 818,741 entries.
  • Corpus: 1.77 million documents including Newswire and Web Text.

Results and Discussion

Our experiments demonstrate that collaborative ranking significantly outperforms standalone baselines. The best standalone supervised ranker, ListNet, achieved 81.1% accuracy. By applying MiCR to a simple TF-IDF ranker, we observed a 5.3% absolute improvement. MaCR provided a further 1.3% gain over the best baseline.

The comprehensive MiMaCR approach reached a peak accuracy of 83.7%, a 2.6% absolute improvement over the baseline. This performance is statistically significant and competitive with the top-tier systems in the KBP 2010 international evaluation.

Key insights include:

  • The "Support Group" Effect: Collaborators provide multifaceted context (e.g., one document mentions a player's jersey number, another their team), making disambiguation easier.
  • Clustering Strategy: Graph-based clustering proved superior to agglomerative methods by effectively filtering out noisy collaborators.
  • Diversity in MaCR: The effectiveness of macro-collaboration depends on the diversity of the underlying rankers. The authors noted that supervised rankers using similar features yielded smaller gains compared to ensembles of diverse systems.

Conclusion

Collaborative Ranking offers a powerful framework for improving search and disambiguation tasks. While MiMaCR introduces higher computational costs due to clustering and multiple ranking passes, the accuracy gains are substantial. Future work will explore applying this framework to other ranking-heavy tasks such as Question Answering or Machine Translation.

Summary updated

Abstract

In this paper, we present a new ranking scheme, collaborative ranking (CR). In contrast to traditional non-collaborative ranking scheme which solely relies on the strengths of isolated queries and one stand-alone ranking algorithm, the new scheme integrates the strengths from multiple collaborators of a query and the strengths from multiple ranking algorithms. We elaborate three specific forms of collaborative ranking, namely, micro collaborative ranking (MiCR), macro collaborative ranking (MaCR) and micro-macro collaborative ranking (MiMaCR). Experiments on entity linking task show that our proposed scheme is indeed effective and promising.
BibTeX
@inproceedings{chen2011collaborative,
  author = {Zheng Chen and Heng Ji},
  title = {{Collaborative Ranking: A Case Study on Entity Linking}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2011},
  url = {https://openreview.net/forum?id=r1VwR-MdZB},
}
Figure