TL;DR
While Transformers completely dominate sequence labeling in NLP, this paper systematically evaluates how emerging alternative architectures—such as xLSTMs, Mamba, diffusion models, and adversarial networks—perform across tasks of varying structural complexity. The authors find that while adversarial tagging consistently rivals Transformers, state-space models and diffusion taggers struggle significantly when tasks require tracking complex, long-range structural dependencies.
What Problem Does This Paper Address?
Sequence labeling (SL) is a foundational task in NLP where every token in a sentence is assigned a label (e.g., Part-of-Speech tagging or Named Entity Recognition). By using clever "linearization" strategies, even complex hierarchical tasks like syntactic dependency and graph parsing can be mapped into a sequence of tags.
Currently, pretrained Transformer encoders are the undisputed kings of sequence labeling. Meanwhile, alternative architectures like xLSTMs, structured state-space models (SSMs like Mamba), diffusion models, and Generative Adversarial Networks (GANs) have shown massive promise in language modeling or computer vision. However, their efficacy in sequence labeling has only been tested on simple, "flat" tasks. This paper addresses this gap by benchmarking these emerging architectures on a unified, highly demanding multilingual framework to see if their success truly generalizes.
Methods
The authors adapt and evaluate four distinct architectural paradigms for sequence labeling:
- Diffusion Tagging (DiT): Uses a Bit-Tag converter to map discrete labels into continuous binary bit vectors. A conditional diffusion model (utilizing DDIM sampling) is then trained to iteratively denoise a random Gaussian signal into the correct bit-sequence representation of the tags.
- Adversarial Tagging (GaT): A GAN-style framework where a Transformer-based generator predicts tag distributions, and a lightweight 2-layer BiLSTM discriminator evaluates whether the sequence matches the gold-standard distribution. Gumbel-Softmax relaxation is used to keep the discrete outputs differentiable.
- xLSTM (BixLSTM): Utilizes the recently proposed xLSTM architecture, which introduces exponential gating and matrix memory structures to overcome traditional LSTM limitations. The authors test a bidirectional variant (BixLSTM).
- MAMBA-2 (SSD): A state-of-the-art structured state-space model designed for linear-time sequence modeling, evaluated here as an alternative encoder backbone.
Datasets
The paper utilizes a massive, multilingual benchmark spanning several tasks:
- Part-of-Speech (PoS) Tagging: Evaluated on 8 Universal Dependencies (UD) treebanks, the Penn Treebank (PTB), the Chinese Treebank (CTB), and the Statistical Parsing of Morphologically Rich Languages (SPMRL) datasets.
- Named Entity Recognition (NER): 12 distinct datasets spanning multiple languages and domains, including CoNLL-2003, BioNLP, DrugsNER, GermEval-2014, and WikiNER.
- Linearized Parsing: Syntactic constituency, dependency, and semantic graph parsing tasks from the PTB, CTB, UD, and IWPT shared tasks, all converted into token-level tag sequences.
Key Insights and Ideas
- Structure is the Ultimate Filter: Alternative architectures like Mamba-2 and Diffusion models perform competitively on simple, flat tasks (like basic PoS tagging). However, their performance collapses on structurally complex tasks like graph and dependency parsing.
- The Adversarial Advantage: Adversarial training (GaT) acts as an excellent structural regularizer. By forcing the generator to fool a discriminator, the model learns to internalize global syntactic constraints, producing significantly more "well-formed" and valid parsing sequences than standard baselines.
- Bidirectionality is Mandatory: Unidirectional models (standard xLSTM and Mamba-2) suffer severe performance decay when modeling long-range dependencies in parsing. For sequence labeling, processing text from both left-to-right and right-to-left remains non-negotiable.
Key Results
- Adversarial Tagging Wins: The GaT model consistently outperformed all other non-baseline models, achieving the best scores in 20 out of 22 PoS tagging setups. Crucially, it was the only alternative model to rival or slightly exceed the Transformer baseline on complex parsing tasks (e.g., achieving 87.70 vs. 87.49 LAS in dependency parsing).
- BixLSTM is a Strong Non-Pretrained Alternative: BixLSTM outperformed traditional BiLSTMs in 19 out of 22 PoS tagging tasks. While it still trails pretrained Transformers, it offers an excellent accuracy-to-speed tradeoff for scenarios where pretraining isn't viable.
- Mamba and Diffusion Struggle: Mamba-2 struggled heavily with complex token dependencies, performing on par with simple unidirectional LSTMs. Diffusion tagging failed on highly unbalanced datasets (like Weibo NER) due to its reliance on Mean Squared Error (MSE) loss over continuous bit spaces.
The trade-offs between accuracy (LAS) and inference speed (sentences per second) across these architectures are clearly illustrated in the paper's Pareto front analysis:
Limitations and Future Work
- Masked Language Models Only: Due to computational constraints and the fact that autoregressive sequence tagging historically lags behind bidirectional encoders, the authors limited their Transformer baselines to Masked Language Models (MLMs like XLM-RoBERTa). Evaluating how these architectures compare against massive decoder-only LLMs remains open.
- Scale Limitations: The alternative models were evaluated at relatively small scales due to academic hardware constraints (RTX 3090 and A100 GPUs).
- Future Directions: The authors suggest exploring how architectures like xLSTM and Mamba perform when scaled up with massive pretraining datasets specifically tailored for token-level classification.
Summary updated