Abstract
In this paper, we present a new ranking scheme, collaborative ranking (CR). In contrast to traditional non-collaborative ranking scheme which solely relies on the strengths of isolated queries and one stand-alone ranking algorithm, the new scheme integrates the strengths from multiple collaborators of a query and the strengths from multiple ranking algorithms. We elaborate three specific forms of collaborative ranking, namely, micro collaborative ranking (MiCR), macro collaborative ranking (MaCR) and micro-macro collaborative ranking (MiMaCR). Experiments on entity linking task show that our proposed scheme is indeed effective and promising.
Introduction
Traditional ranking systems often treat individual queries as isolated events. In tasks such as Entity Linking (EL)—where a mention like "Michael Jordan" must be mapped to a specific entry in a Knowledge Base (KB)—this isolation leads to two primary issues: information scarcity and model bias. A single document may lack sufficient context to disambiguate an entity, and no single ranking algorithm (pointwise, pairwise, or listwise) is universally optimal.
To address these limitations, we propose Collaborative Ranking (CR). This framework leverages "collaborators" across two dimensions: the micro-level (similar queries or documents) and the macro-level (diverse ranking algorithms). By aggregating information from these sources, the system can make more robust ranking decisions.
Methodology: Three Forms of Collaborative Ranking
We define three distinct implementations of the CR framework:
- Micro Collaborative Ranking (MiCR): Focuses on query-level collaboration. For a target query, the system identifies "collaborators" (other documents in a large corpus mentioning the same name string). Using graph-based or agglomerative clustering, these documents are grouped. The final ranking score for a candidate object is an aggregate (e.g., max, min, or average) of scores from the query and its collaborators.
- Macro Collaborative Ranking (MaCR): Focuses on ranker-level collaboration. This is an ensemble approach that integrates multiple ranking functions (e.g., TF-IDF and supervised ListNet) using composite functions like voting or weighted averaging.
- Micro-Macro Collaborative Ranking (MiMaCR): A hybrid approach that first computes micro-ranking scores for each individual algorithm across query collaborators, then performs a macro-integration across the different algorithms.
Experimental Setup
The proposed schemes were evaluated on the Entity Linking task using the TAC-KBP (Text Analysis Conference - Knowledge Base Population) tracks:
- Data: TAC-KBP 2009 (3,904 queries) for training/dev and TAC-KBP 2010 (2,250 queries) for testing.
- Knowledge Base: A snapshot of English Wikipedia (Oct 2008) containing approximately 818,741 entries.
- Corpus: 1.77 million documents including Newswire and Web Text.
Results and Discussion
Our experiments demonstrate that collaborative ranking significantly outperforms standalone baselines. The best standalone supervised ranker, ListNet, achieved 81.1% accuracy. By applying MiCR to a simple TF-IDF ranker, we observed a 5.3% absolute improvement. MaCR provided a further 1.3% gain over the best baseline.
The comprehensive MiMaCR approach reached a peak accuracy of 83.7%, a 2.6% absolute improvement over the baseline. This performance is statistically significant and competitive with the top-tier systems in the KBP 2010 international evaluation.
Key insights include:
- The "Support Group" Effect: Collaborators provide multifaceted context (e.g., one document mentions a player's jersey number, another their team), making disambiguation easier.
- Clustering Strategy: Graph-based clustering proved superior to agglomerative methods by effectively filtering out noisy collaborators.
- Diversity in MaCR: The effectiveness of macro-collaboration depends on the diversity of the underlying rankers. The authors noted that supervised rankers using similar features yielded smaller gains compared to ensembles of diverse systems.
Conclusion
Collaborative Ranking offers a powerful framework for improving search and disambiguation tasks. While MiMaCR introduces higher computational costs due to clustering and multiple ranking passes, the accuracy gains are substantial. Future work will explore applying this framework to other ranking-heavy tasks such as Question Answering or Machine Translation.
Summary updated