When Generative Adversarial Networks Meet Sequence Labeling Challenges

Yu Tong, Ge Chen, Guokai Zheng, Rui Li, Jiang DazhiEMNLP 2024 · DOI

Abstract

Sequence labeling tasks such as Named Entity Recognition (NER) and word segmentation often struggle with out-of-vocabulary (OOV) words and distribution disparities between text and labels. We present SLGAN (Sequence Labeling GAN), a unified framework that adapts Generative Adversarial Networks to discrete sequence labeling. By employing a "bystander" discriminator that evaluates individual tag correctness and a Gumbel-Softmax layer for differentiability, SLGAN achieves state-of-the-art performance across 14 datasets in four languages, showing significant improvements in OOV recall.

Introduction

Sequence labeling—the process of assigning a tag to every token in a string—is fundamental to NLP. Despite the success of Generative Adversarial Networks (GANs) in continuous domains like computer vision, their application to text has been hindered by the discrete nature of language. Traditional backpropagation cannot easily "nudge" discrete tokens or tags. Furthermore, existing models often fail when encountering words unseen during training (the OOV problem) and struggle with the inherent distribution disparities between natural language text and its corresponding label sequences.

Methodology

The SLGAN framework addresses these challenges through a novel generator-discriminator architecture designed specifically for sequence labeling.

The SLGAN architecture featuring a BERT-based generator and a dual-stream attention discriminator.

The Generator (G)

The generator utilizes a standard sequence labeling stack: a BERT encoder followed by fully connected layers and a Conditional Random Field (CRF) layer. It takes an input sentence and predicts the most likely sequence of tags.

The Discriminator (D)

Unlike traditional GAN discriminators that provide a global "real vs. fake" judgment, the SLGAN discriminator acts as a "bystander" critic. It evaluates the correctness of each individual tag within the context of the sentence.

Technical Innovations

  • Gumbel-Softmax Layer: To bridge the gap between discrete tags and differentiable gradients, a Gumbel-Softmax trick is employed. This provides a continuous approximation of the tag distribution, allowing gradients to flow from the discriminator back to the generator.
  • Dual-Stream Attention: To handle the different distributions of text and labels, the discriminator uses an attention mechanism to fuse BERT-encoded text features with label features, capturing the contextual appropriateness of each tag.
  • Composite Objective: The generator is trained using a combined loss function that includes the standard supervised CRF loss and adversarial feedback from the discriminator.

Experimental Evaluation

SLGAN was evaluated across 14 datasets covering four languages (Chinese, Japanese, Korean, and English) and various tasks including Chinese Word Segmentation (CWS), Named Entity Recognition (NER), and Part-of-Speech (PoS) tagging.

Results and Discussion

SLGAN achieved state-of-the-art (SOTA) or highly competitive results across nearly all benchmarks. A key strength of the model is its robustness to out-of-vocabulary words.

Performance of SLGAN on Chinese Word Segmentation datasets, highlighting F1 scores and OOV recall (R_oov).

As shown in the results for Chinese Word Segmentation (CWS), SLGAN consistently outperforms strong baselines like BERT-CRF. Notably, on the AS dataset, the model achieves a significant boost in OOV recall (R_oov). The adversarial training allows the discriminator to learn the semantic relationships between context and tags, enabling the generator to generalize better to rare entities and neologisms.

Beyond standard benchmarks, SLGAN demonstrated exceptional cross-domain performance. For instance, on the "Dermatology" (DM) medical dataset, it outperformed previous SOTA results by +2.79% in F1 score without the aid of domain-specific dictionaries. The framework's language-agnostic nature was further proven by its SOTA performance on English NER (CoNLL2003) and Korean word segmentation.

Conclusion and Future Work

SLGAN successfully bridges the gap between GANs and sequence labeling. By providing granular feedback through a token-level discriminator and solving differentiability issues with Gumbel-Softmax, the framework significantly improves model robustness and OOV handling. While GAN training remains more complex and potentially unstable compared to supervised methods, the performance gains across diverse languages and domains suggest that adversarial learning is a powerful tool for discrete sequence tasks. Future work may explore extending this framework to more complex structural tasks like dependency parsing.

Summary updated

BibTeX
@inproceedings{tong2024when,
  author = {Yu Tong and Ge Chen and Guokai Zheng and Rui Li and Jiang Dazhi},
  title = {{When Generative Adversarial Networks Meet Sequence Labeling Challenges}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.593},
}
Figure