MGAD: Learning Descriptional Representation Distilled from Distributional Semantics for Unseen Entities

Yuanzheng Wang, Xueqi Cheng, Yixing Fan, Xiaofei Zhu, Huasheng Liang, Qiang Yan, Jiafeng Guo

Abstract

Entity retrieval often struggles with "unseen" entities—those present in a knowledge base with only a description but no historical context. We present MGAD (Multi-Granularity Alignments based Distillation), a framework that transfers semantic knowledge from context-rich distributional models to description-based student models. By employing three levels of spatial alignment—element, batch, and space—MGAD enables student models to "hallucinate" distributional properties from text descriptions alone, achieving state-of-the-art performance on both seen and unseen entities.

Introduction

In Natural Language Processing, Entity Retrieval links mentions in text to specific Knowledge Base (KB) entries. Existing models typically fall into two categories:

  1. Descriptional models: Represent entities via names and official descriptions. They are flexible but often lack the semantic depth of context-aware models.
  2. Distributional models: Learn representations from the contexts in which entities appear. While highly accurate for frequent entities, they suffer from a "cold-start" problem, failing on "unseen" entities that lack historical context in training data.

MGAD addresses this gap by distilling the "semantic intuition" of a distributional teacher into a descriptional student.

The MGAD Framework

The core of MGAD is a Knowledge Distillation process where a high-performing distributional model (e.g., LUKE) acts as the teacher, and a description-based model (e.g., BLINK/RoBERTa) acts as the student.

Overview of the MGAD framework showing the alignment between descriptional and distributional embedding spaces.

Multi-Granularity Alignment

To bridge the gap between the student’s descriptional space ($S_{desc}$) and the teacher’s distributional space ($S_{dist}$), MGAD introduces three distinct alignment losses:

  • Element-level ($L_{ea}$): A "one-to-one" alignment that forces the student's description embedding to match the teacher's context-based embedding for the same entity.
  • Batch-level ($L_{ba}$): A "one-to-batch" alignment ensuring the student’s probability distribution over a candidate batch matches the teacher’s.
  • Space-level ($L_{sa}$): A "one-to-all" alignment where the student must identify the correct teacher embedding from the entire global space of seen entities.
Illustration of the three alignment losses: (a) element-level one-to-one, (b) batch-level one-to-batch, and (c) space-level one-to-all.

Experimental Setup

The model was evaluated on several widely-used entity linking benchmarks, including AIDA (CoNLL-2003) for in-domain testing, and ACE, MSNBC, and AQUAINT for out-of-domain generalization. The entity collection was derived from Wikipedia, where the 500k most popular entities were treated as "seen" (for the teacher) and others as "unseen."

Key Results and Insights

MGAD demonstrates significant improvements in handling the cold-start problem for entity retrieval:

  • Superiority on Unseen Entities: On AIDA-testB, MGAD achieved a 95.51 MRR, outperforming BLINK (94.19) and GENRE (87.41).
  • Closing the Teacher-Student Gap: MGAD performs nearly as well as its teacher on "seen" entities, despite the teacher having access to context that the student lacks at inference time.
  • Robustness: Unlike distributional models, MGAD maintains high performance regardless of entity frequency.
  • Ablation Study: Space-level alignment ($L_{sa}$) proved to be the most critical component, providing a stronger global signal than local batch alignments.

Discussion and Future Work

While MGAD excels at retrieval, the authors noted that entities with "intermediate" frequency (100–1,000 mentions) showed the most ambiguity, proving harder to rank than very frequent or very rare entities. Additionally, as MGAD focuses on the candidate generation (retrieval) stage, future work could apply this distillation logic to the subsequent "re-ranking" stage of entity linking to further boost accuracy.

Conclusion

MGAD successfully transfers the semantic richness of distributional models to description-based encoders. By aligning representations across multiple granularities, it provides a robust solution for retrieving rare or newly emerging entities that lack extensive historical context.

Summary updated

Abstract

Entity representation plays a central role in building effective entity retrieval models. Recent works propose to learn entity representations based on entity-centric contexts, which achieve SOTA performances on many tasks. However, these methods lead to poor representations for unseen entities since they rely on a multitude of occurrences for each entity to enable accurate representations. To address this issue, we propose to learn enhanced descriptional representations for unseen entities by distilling knowledge from distributional semantics into descriptional embeddings. Specifically, we infer enhanced embeddings for unseen entities based on descriptions by aligning the descriptional embedding space to the distributional embedding space with different granularities, i.e., element-level, batch-level and space-level alignment. Experimental results on four benchmark datasets show that our approach improves the performance over all baseline methods. In particular, our approach can achieve the effectiveness of the teacher model on almost all entities, and maintain such high performance on unseen entities.
BibTeX
@inproceedings{wang2022mgad,
  author = {Yuanzheng Wang and Xueqi Cheng and Yixing Fan and Xiaofei Zhu and Huasheng Liang and Qiang Yan and Jiafeng Guo},
  title = {{MGAD: Learning Descriptional Representation Distilled from Distributional Semantics for Unseen Entities}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/611},
}
Figure