Counterfactual Generator: A Weakly-Supervised Method for Named Entity Recognition

Xiangji Zeng, Yunliang Li, Yuchen Zhai, Yin ZhangEMNLP 2020 · OpenReview

Abstract

Xiangji Zeng, Yunliang Li, Yuchen Zhai, Yin Zhang. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

2020.

Introduction

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing, involving the identification and categorization of entities such as medical diagnoses or geographic locations. However, training effective NER models typically requires extensive human-annotated datasets, which are costly and difficult to obtain in specialized domains like medicine.

In low-resource scenarios, models often rely on "spurious correlations"—statistical shortcuts where the model associates specific entities with their surrounding contexts rather than learning the intrinsic features of the entities themselves. For instance, a model might incorrectly label any word following "biopsy" as a disease because of its frequent association in the training set. This paper introduces a Counterfactual Generator to break these shortcuts by disentangling entities from their contexts.

Methodology: The Counterfactual Generator

The proposed framework employs a causal approach to data augmentation, specifically through entity intervention. The process consists of three primary stages:

  1. Set Preparation: Entities are extracted from the limited training data and categorized into "buckets" based on their types (e.g., all "Diagnosis" entities are grouped together).
  2. Entity Intervention: For a given sentence, an entity is randomly replaced with another entity of the same type from the prepared buckets. This creates a "counterfactual" scenario—a "what-if" sentence that maintains the same context but changes the entity.
  3. Example Discrimination: To ensure linguistic and logical consistency, a base model (trained on the original data) acts as a discriminator. Only counterfactual examples that the model can still accurately identify are added to the augmented training set.
The intervention process swaps entities between contexts to break spurious correlations.

By exposing the model to the same context with various entities (and vice versa), the framework forces the model to learn robust, context-independent features.

Theoretical Framework: Structural Causal Models

The Counterfactual Generator is grounded in Structural Causal Models (SCMs). The authors model the relationship between the latent entity representation (E), context representation (C), and the observed sentence (X) leading to the label (Y).

Causal graphs representing (a) the original observational model, and (b, c) interventions on context and entity variables.

Through these models, the researchers identified that context is often the primary source of spuriousness. By intervening on the entity variable (as shown in graph 'c'), they can "mute" misleading signals from the context. Their analysis using the Relative Importance (RI) Index further confirmed that an entity's own representation is more critical for accurate NER than its surrounding context.

Experimental Results

The method was evaluated on three Chinese-language datasets: CNER (clinical records), IDiag (noisy OCR medical data), and CLUENER (fine-grained general entities).

The Counterfactual Generator demonstrated significant performance gains, particularly in low-resource settings:

  • CNER: With only 300 training examples, the model achieved a +15.4% F1 score improvement.
  • IDiag: Despite the noise in OCR data, the method provided an average boost of +5.2%.
  • Generalization: Models trained with counterfactual augmentation performed substantially better on "unseen" entities, indicating improved generalization rather than simple memorization.

Discussion and Limitations

While the Counterfactual Generator effectively "squeezes" more information out of limited labels, it faces certain challenges. The quality of generated data is bounded by the Discriminator Bottleneck; if the initial base model is weak, the filter may fail to remove poor-quality examples. Additionally, the process can lead to high text repetition, which may occasionally hinder the convergence of Conditional Random Field (CRF) layers. Finally, while swapping entities works for NER, it may disrupt the factual logic of a sentence, potentially limiting its application in tasks requiring deep semantic reasoning like Natural Language Inference.

Summary updated

BibTeX
@inproceedings{zeng2020counterfactual,
  author = {Xiangji Zeng and Yunliang Li and Yuchen Zhai and Yin Zhang},
  title = {{Counterfactual Generator: A Weakly-Supervised Method for Named Entity Recognition}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=Mb7-LQQJsJJ},
}
Figure