GNNer: Reducing Overlapping in Span-based NER Using Graph Neural Networks

Urchade Zaratiana, Nadi Tomeh, Pierre Holat, Thierry CharnoisACL 2022

TL;DR

This paper introduces GNNer, a novel framework that leverages Graph Neural Networks (GNNs) to reduce overlapping entity predictions in span-based Named Entity Recognition (NER). By constructing a graph of overlapping spans and using GNNs to enrich span representations, GNNer implicitly enforces a no-overlap constraint during training, leading to fewer overlapping entities while maintaining competitive performance.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a crucial task in information extraction, aiming to identify and categorize named entities (like people, organizations, and locations) within text. While sequence labeling approaches have been dominant, span-based NER, which treats entire spans of text as units, has gained popularity. A key challenge with span-based NER is its tendency to predict overlapping entities, which is often undesirable, especially in "flat" NER tasks where entities are not expected to nest. Existing methods to address this either rely on post-processing heuristic decoding or impose constraints during the decoding phase. This paper argues that imposing the no-overlap constraint directly during the learning phase, by injecting inductive biases into the model, is a more optimal approach.

Methods

The GNNer framework builds upon a standard span-based NER architecture by introducing a GNN layer. The core components are:

  1. Token Representation Layer: Uses pre-trained Transformer models (like BERT or SciBERT) to generate contextualized embeddings for each word. For words that are split into sub-words by the tokenizer, the first sub-word's representation is used.
  2. Span Representation Layer: Spans are enumerated up to a maximum width (set to 6 in experiments). Each span's representation is formed by concatenating the embeddings of its start and end words, along with a learned embedding for the span's width.
  3. Graph Construction: A graph is built where nodes represent the enumerated spans. An adjacency matrix defines the edges:
  4. Weight 1 for self-connections (a span with itself).
  5. Weight 0 for non-overlapping spans.
  6. Weight -1 for overlapping spans. The negative weight is intended to encourage the model to learn dissimilar representations for overlapping spans.
  7. Span Refinement with GNN: This is the novel part. The span representations are refined using a GNN layer. Two variants are proposed:
  8. GNNer-CONV: Uses a Graph Convolutional Network (GCN). Since GCNs can struggle with negative edges, two independent GCNs are run: one on positive edges (self-connections) and another on negative edges (overlaps). The outputs are concatenated.
  9. GNNer-AT: Uses a Graph Attention Network (GAT) with dot-product attention. This variant is noted to be more computationally expensive due to quadratic complexity with the number of spans.
  10. Span Classification Layer: The refined span representations are fed into a linear layer with a softmax activation to predict the final span labels (including a null label for non-entities).

Datasets

The paper evaluates GNNer on three benchmark datasets:

  • Conll-2003: A general domain NER dataset.
  • SciERC NER: Focuses on scientific information extraction from AI-related articles.
  • NCBI: A dataset for identifying disease mentions in biomedical texts.

The standard train, validation, and test splits were used for all datasets.

Key Insights and Ideas

  • Implicit Constraint Learning: The core idea is that by explicitly modeling the relationships (overlaps) between spans in a graph and using GNNs, the model can learn to avoid predicting overlapping entities implicitly during training, rather than relying on explicit rules during decoding.
  • Negative Edges as a Signal: The use of negative edges in the graph construction is a deliberate choice to push the model towards learning distinct representations for overlapping spans, thereby discouraging their simultaneous prediction.
  • Trade-offs in GNN Variants: The GNNer-CONV and GNNer-AT variants offer different trade-offs. GNNer-AT is more effective at reducing overlaps but at the cost of recall, while GNNer-CONV provides a better balance between overlap reduction and overall performance.

Key Results

  • Overlap Reduction: GNNer-AT significantly reduces the number of overlapping spans (Num. Ov.), outperforming the baseline by approximately 4 times and GNNer-CONV by 2 times. GNNer-CONV also shows substantial overlap reduction compared to the baseline.
  • Performance Balance: GNNer-CONV achieves competitive precision, recall, and F1-scores while significantly reducing overlaps, making it the preferred choice for a balanced approach.
  • Recall Impact: GNNer-AT's aggressive overlap reduction comes at the cost of lower recall, suggesting that the strong no-overlap signal might sometimes prevent the model from identifying valid entities.
  • Training Dynamics: Learning curves show that GNN-based methods might train slower than the baseline, potentially due to the added complexity of the GNN layer and the implicit constraint.

Limitations and Future Work

  • Incomplete Overlap Removal: The authors acknowledge that GNNer does not completely eliminate overlapping spans, unlike some heuristic approaches.
  • Computational Cost: The GNN layer adds computational overhead, making the model slower than the baseline, especially since the span graph can be dense, limiting the benefits of efficient sparse GNN operations.
  • Future Work: The authors suggest exploring ways to integrate GNN-enhanced representations into architectures for joint named entity recognition and relation extraction tasks. They also note that further investigation into the optimal choice for edge weights in the span graph could be beneficial.

Summary updated

BibTeX
@inproceedings{zaratiana2022gnner,
  author = {Urchade Zaratiana and Nadi Tomeh and Pierre Holat and Thierry Charnois},
  title = {{GNNer: Reducing Overlapping in Span-based NER Using Graph Neural Networks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure