Abstract
Sequence Labeling (SL) is a fundamental task in Natural Language Processing (NLP). Traditionally, discriminative models have been widely used to capture the conditional distribution of sequence tags. In this paper, we present DiffusionSL, a framework that utilizes a conditional discrete diffusion model for generating discrete tag data, resulting in a Tag Diffusion Process. We treat the natural language sequence as the conditional signal and the sequence tags as the generation target, iteratively refining noisy tags to obtain clean ones. To address the discreteness issue, we propose the Bit-Tag Converter (BTConverter) to model the target in continuous data space and the Bit Diffusion Transformer (BitDiT) to model the noise elimination process. DiffusionSL achieves superior performance against previous state-of-the-art (SOTA) baselines and significantly outperforms gpt-3.5-turbo across multiple benchmark datasets.
Introduction
Sequence Labeling (SL)—assigning a tag to every unit in a sentence—powers essential NLP applications like Named Entity Recognition (NER) and Part-of-Speech (POS) tagging. Historically, two approaches dominated:
- Discriminative Models: Efficient but often limited in capturing complex global dependencies between tags.
- Autoregressive Generative Models: Flexible but prone to "exposure bias," where early errors cascade through the sequence because the model is not trained on its own mistakes during inference.
DiffusionSL addresses these gaps by reimagining SL as a generative diffusion process. Unlike autoregressive models, it predicts all tags simultaneously and iteratively refines them, allowing the model to "fix" its mistakes using global context.
The DiffusionSL Framework
The core of DiffusionSL is the Tag Diffusion Process, which treats sequence labeling as a conditional generation task. The framework relies on two primary innovations to bridge the gap between continuous diffusion mathematics and discrete linguistic tags:
Bit-Tag Converter (BTConverter)
Standard diffusion models operate in continuous spaces (like pixel values). To apply this to discrete tags, BTConverter represents each tag as a sequence of binary bits (e.g., tag #5 becomes 0101). These bits are treated as real numbers in a continuous range, allowing the model to add Gaussian noise to and subsequently denoise the tags.
Bit Diffusion Transformer (BitDiT)
BitDiT serves as the denoising engine. Unlike standard Transformers that rely on cross-attention, BitDiT utilizes Adaptive Layer Normalization (AdaLN). This mechanism injects the conditional signal (the input sentence and the current time step) directly into the normalization layers, leading to faster convergence and more robust performance.
Experimental Evaluation
DiffusionSL was evaluated on seven benchmark datasets across three major tasks:
- Named Entity Recognition (NER): MSRA, Resume, and CoNLL03.
- Chinese Word Segmentation (CWS): MSRA, PKU, and CTB6.
- Part-of-Speech (POS) Tagging: CTB5.
Key Results
- State-of-the-Art Performance: DiffusionSL established new SOTA results, achieving F1 scores of 95.49 on MSRA (NER) and 98.33 on MSRA (CWS).
- Efficiency: The model requires only 10 denoising steps to reach peak performance, balancing accuracy with inference speed.
- Convergence: The BitDiT architecture converges significantly faster than traditional cross-attention decoders.
- Comparison to LLMs: DiffusionSL consistently outperformed gpt-3.5-turbo, demonstrating that specialized diffusion models remain superior for structured prediction tasks.
Discussion and Insights
One of the most surprising findings was that representing tags as binary bits performed significantly better than using learnable continuous embeddings. This suggests that bit representation provides a clearer, more lightweight "map" for the diffusion process to follow. Furthermore, the non-autoregressive nature of the model avoids the linear time complexity of token-by-token generation, making it highly suitable for long sequences.
Limitations
While powerful, DiffusionSL faces higher inference latency than simple "one-pass" discriminative taggers (like BERT) due to its iterative nature. Additionally, the BitDiT decoder requires more precise hyperparameter tuning than established architectures. Future work will focus on extending this framework to more complex tasks like Semantic Role Labeling (SRL).
Summary updated