Abstract
Entity retrieval often struggles with "unseen" entities—those present in a knowledge base with only a description but no historical context. We present MGAD (Multi-Granularity Alignments based Distillation), a framework that transfers semantic knowledge from context-rich distributional models to description-based student models. By employing three levels of spatial alignment—element, batch, and space—MGAD enables student models to "hallucinate" distributional properties from text descriptions alone, achieving state-of-the-art performance on both seen and unseen entities.
Introduction
In Natural Language Processing, Entity Retrieval links mentions in text to specific Knowledge Base (KB) entries. Existing models typically fall into two categories:
- Descriptional models: Represent entities via names and official descriptions. They are flexible but often lack the semantic depth of context-aware models.
- Distributional models: Learn representations from the contexts in which entities appear. While highly accurate for frequent entities, they suffer from a "cold-start" problem, failing on "unseen" entities that lack historical context in training data.
MGAD addresses this gap by distilling the "semantic intuition" of a distributional teacher into a descriptional student.
The MGAD Framework
The core of MGAD is a Knowledge Distillation process where a high-performing distributional model (e.g., LUKE) acts as the teacher, and a description-based model (e.g., BLINK/RoBERTa) acts as the student.
Multi-Granularity Alignment
To bridge the gap between the student’s descriptional space ($S_{desc}$) and the teacher’s distributional space ($S_{dist}$), MGAD introduces three distinct alignment losses:
- Element-level ($L_{ea}$): A "one-to-one" alignment that forces the student's description embedding to match the teacher's context-based embedding for the same entity.
- Batch-level ($L_{ba}$): A "one-to-batch" alignment ensuring the student’s probability distribution over a candidate batch matches the teacher’s.
- Space-level ($L_{sa}$): A "one-to-all" alignment where the student must identify the correct teacher embedding from the entire global space of seen entities.
Experimental Setup
The model was evaluated on several widely-used entity linking benchmarks, including AIDA (CoNLL-2003) for in-domain testing, and ACE, MSNBC, and AQUAINT for out-of-domain generalization. The entity collection was derived from Wikipedia, where the 500k most popular entities were treated as "seen" (for the teacher) and others as "unseen."
Key Results and Insights
MGAD demonstrates significant improvements in handling the cold-start problem for entity retrieval:
- Superiority on Unseen Entities: On AIDA-testB, MGAD achieved a 95.51 MRR, outperforming BLINK (94.19) and GENRE (87.41).
- Closing the Teacher-Student Gap: MGAD performs nearly as well as its teacher on "seen" entities, despite the teacher having access to context that the student lacks at inference time.
- Robustness: Unlike distributional models, MGAD maintains high performance regardless of entity frequency.
- Ablation Study: Space-level alignment ($L_{sa}$) proved to be the most critical component, providing a stronger global signal than local batch alignments.
Discussion and Future Work
While MGAD excels at retrieval, the authors noted that entities with "intermediate" frequency (100–1,000 mentions) showed the most ambiguity, proving harder to rank than very frequent or very rare entities. Additionally, as MGAD focuses on the candidate generation (retrieval) stage, future work could apply this distillation logic to the subsequent "re-ranking" stage of entity linking to further boost accuracy.
Conclusion
MGAD successfully transfers the semantic richness of distributional models to description-based encoders. By aligning representations across multiple granularities, it provides a robust solution for retrieving rare or newly emerging entities that lack extensive historical context.
Summary updated