DiffusionSL: Sequence Labeling via Tag Diffusion Process

Ziyang Huang, Pengfei Cao, Jun Zhao, Kang LiuEMNLP 2023 · OpenReview

Abstract

Sequence Labeling (SL) is a fundamental task in Natural Language Processing (NLP). Traditionally, discriminative models have been widely used to capture the conditional distribution of sequence tags. In this paper, we present DiffusionSL, a framework that utilizes a conditional discrete diffusion model for generating discrete tag data, resulting in a Tag Diffusion Process. We treat the natural language sequence as the conditional signal and the sequence tags as the generation target, iteratively refining noisy tags to obtain clean ones. To address the discreteness issue, we propose the Bit-Tag Converter (BTConverter) to model the target in continuous data space and the Bit Diffusion Transformer (BitDiT) to model the noise elimination process. DiffusionSL achieves superior performance against previous state-of-the-art (SOTA) baselines and significantly outperforms gpt-3.5-turbo across multiple benchmark datasets.

Introduction

Sequence Labeling (SL)—assigning a tag to every unit in a sentence—powers essential NLP applications like Named Entity Recognition (NER) and Part-of-Speech (POS) tagging. Historically, two approaches dominated:

  1. Discriminative Models: Efficient but often limited in capturing complex global dependencies between tags.
  2. Autoregressive Generative Models: Flexible but prone to "exposure bias," where early errors cascade through the sequence because the model is not trained on its own mistakes during inference.
Comparison of Autoregressive generation (a) vs. Non-Autoregressive Diffusion refinement (b). The diffusion process allows the model to correct initial errors (red X) into accurate tags (green check).

DiffusionSL addresses these gaps by reimagining SL as a generative diffusion process. Unlike autoregressive models, it predicts all tags simultaneously and iteratively refines them, allowing the model to "fix" its mistakes using global context.

The DiffusionSL Framework

The core of DiffusionSL is the Tag Diffusion Process, which treats sequence labeling as a conditional generation task. The framework relies on two primary innovations to bridge the gap between continuous diffusion mathematics and discrete linguistic tags:

Bit-Tag Converter (BTConverter)

Standard diffusion models operate in continuous spaces (like pixel values). To apply this to discrete tags, BTConverter represents each tag as a sequence of binary bits (e.g., tag #5 becomes 0101). These bits are treated as real numbers in a continuous range, allowing the model to add Gaussian noise to and subsequently denoise the tags.

Bit Diffusion Transformer (BitDiT)

BitDiT serves as the denoising engine. Unlike standard Transformers that rely on cross-attention, BitDiT utilizes Adaptive Layer Normalization (AdaLN). This mechanism injects the conditional signal (the input sentence and the current time step) directly into the normalization layers, leading to faster convergence and more robust performance.

The complete DiffusionSL architecture, illustrating (a) the Tag Diffusion Process from noise to clean tags, (b) the Bit-Tag Converter, and (c) the Bit Diffusion Transformer.

Experimental Evaluation

DiffusionSL was evaluated on seven benchmark datasets across three major tasks:

  • Named Entity Recognition (NER): MSRA, Resume, and CoNLL03.
  • Chinese Word Segmentation (CWS): MSRA, PKU, and CTB6.
  • Part-of-Speech (POS) Tagging: CTB5.

Key Results

  • State-of-the-Art Performance: DiffusionSL established new SOTA results, achieving F1 scores of 95.49 on MSRA (NER) and 98.33 on MSRA (CWS).
  • Efficiency: The model requires only 10 denoising steps to reach peak performance, balancing accuracy with inference speed.
  • Convergence: The BitDiT architecture converges significantly faster than traditional cross-attention decoders.
  • Comparison to LLMs: DiffusionSL consistently outperformed gpt-3.5-turbo, demonstrating that specialized diffusion models remain superior for structured prediction tasks.

Discussion and Insights

One of the most surprising findings was that representing tags as binary bits performed significantly better than using learnable continuous embeddings. This suggests that bit representation provides a clearer, more lightweight "map" for the diffusion process to follow. Furthermore, the non-autoregressive nature of the model avoids the linear time complexity of token-by-token generation, making it highly suitable for long sequences.

Limitations

While powerful, DiffusionSL faces higher inference latency than simple "one-pass" discriminative taggers (like BERT) due to its iterative nature. Additionally, the BitDiT decoder requires more precise hyperparameter tuning than established architectures. Future work will focus on extending this framework to more complex tasks like Semantic Role Labeling (SRL).

Summary updated

Abstract

Sequence Labeling (SL) is long-standing in Natural Language Processing (NLP). Traditionally, discriminative models have been widely used to capture the conditional distribution of sequence tags, rather than generative models. In this paper, we present DiffusionSL, a framework that utilizes a conditional discrete diffusion model for generating discrete tag data, resulting in a Tag Diffusion Process. We treat the natural language sequence as the conditional signal and the sequence tags as the generation target, iteratively refining the noisy tags to obtain clean ones. To address the discreteness issue, we propose the Bit-Tag Converter (BTConverter) to model the target in continuous data space. Furthermore, we introduce the Bit Diffusion Transformer (BitDiT) to model the process of noise elimination. Leveraging the powerful iterative refinement capability of the diffusion model, DiffusionSL achieves superior performance against previous state-of-the-art (SOTA) baselines and outperforms gpt-3.5-turbo significantly across multiple benchmark datasets and various tasks.
BibTeX
@inproceedings{huang2023diffusionsl,
  author = {Ziyang Huang and Pengfei Cao and Jun Zhao and Kang Liu},
  title = {{DiffusionSL: Sequence Labeling via Tag Diffusion Process}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=BoKg2pcF0H},
}
Figure