Effective Use of Context in Noisy Entity Linking

David Mueller, Greg DurrettEMNLP 2018 · OpenReview

Abstract

Neural entity linking (EL) models often struggle with short, noisy text where context is limited. This paper demonstrates that while complex character-level CNNs fail to capture obvious lexical cues, augmenting an attentive Bi-GRU with simple, sparse lexical features yields a 2.8% absolute improvement, setting a new state-of-the-art on the WikilinksNED dataset.

1. Introduction

Entity Linking is the task of mapping a mention in text to its corresponding entry in a knowledge base (e.g., Wikipedia). While EL is well-established for long-form, clean text, it remains a challenge in "noisy" settings like web snippets or social media. In these cases, context is often limited to fewer than 20 words, and global coherence cues—such as other linked entities in the same document—are typically absent.

Modern neural systems frequently ignore direct word overlaps that are obvious to human readers. For example, if the word "people" appears near a mention, it is a strong signal for an entity like "Limbu People." However, neural models often focus on topically related but non-discriminative words, missing these direct lexical clues.

2. Methodology

The core architecture proposed is an attentive Bi-directional Gated Recurrent Unit (Bi-GRU) model. The model processes the left ($l$) and right ($r$) context of a mention independently. A dot-product attention mechanism is then used to weight context words based on their relevance to a specific candidate entity ($e_t$).

The model architecture utilizes a Bi-GRU with attention, concatenated with sparse lexical features before a final linear layer to produce a score $s_t$.

To improve the model's ability to recognize lexical overlaps, the authors explored two distinct extensions:

  • Character-level CNNs: These were integrated to help the model automatically learn sub-word similarities and correspondences between the context and entity titles.
  • Sparse Lexical Features: These are simple indicator features that fire when a context word is an exact match or a substring of the candidate entity's title. These features are conjoined with the word's relative position (offset) to the mention to provide spatial context.

Training was structured as a multi-class decision among one gold entity and four negative "distractor" titles, which the authors found more effective than standard binary classification tasks.

3. Experimental Results

The model was evaluated on WikilinksNED, a massive benchmark containing 3.2 million entity disambiguation examples. The dataset is notably noisy, with an estimated 35% of examples containing spurious gold labels, which places a practical ceiling on achievable accuracy.

The final model, combining the Bi-GRU, attention, and sparse lexical features, achieved a state-of-the-art accuracy of 75.8% on the test set. This represents a 2.8% absolute improvement over the previous best result of 73.0%. Ablation studies showed that while attention provided a ~1% boost, the addition of sparse features was the primary driver of the performance gain.

4. Analysis: The "Smearing" Effect of CNNs

A significant and surprising finding was that adding character-level CNNs actually decreased performance compared to the base attentive model. Analysis of the attention weights revealed what the authors call a "smearing" effect.

Attention weight comparison between the base model (GRU+ATTN, green) and the CNN-augmented model (blue). The CNN model distributes attention too broadly, failing to focus on critical discriminative words like 'cheese' or 'robot'.

As shown in the visualization, the CNN-augmented model tends to spread its attention across the entire context. By attempting to find sub-word matches everywhere, the model loses the ability to confidently prioritize the most indicative context clues. In contrast, the explicit sparse features provided a strong inductive bias that allowed the model to leverage direct lexical matches without sacrificing focus.

5. Conclusion

This research highlights that for noisy entity linking, the inductive bias provided by manual feature engineering is still superior to current automated character-level modeling. Simple lexical features allow models to reliably identify direct correspondences that complex neural architectures may overlook or "smear." Future work is needed to develop neural architectures that can naturally learn these lexical patterns without reverting to manual feature engineering.

Summary updated

BibTeX
@inproceedings{mueller2018effective,
  author = {David Mueller and Greg Durrett},
  title = {{Effective Use of Context in Noisy Entity Linking}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=S1EUzfG_bS},
}
Figure