Deep Equilibrium Non-Autoregressive Sequence Learning

Zaixiang Zheng, Yi Zhou, Hao Zhou

Abstract

Non-autoregressive (NAR) models offer significant speedups in sequence generation but often lag behind autoregressive (AR) models in quality. This paper introduces DEQNAR, a framework that treats NAR refinement as a dynamical system seeking an equilibrium state. By leveraging Deep Equilibrium (DEQ) networks and root-finding algorithms, DEQNAR solves for the optimal "fixed point" of sequence representations. This approach provides a mathematically grounded alternative to fixed-iteration refinement, resulting in improved stability, constant memory training, and state-of-the-art performance across multiple translation benchmarks.

Introduction

In sequence-to-sequence tasks like machine translation, the standard Transformer (AR) generates tokens sequentially, ensuring high quality at the cost of speed. Non-autoregressive (NAR) models generate tokens in parallel, significantly reducing latency but often sacrificing accuracy. To bridge this gap, many NAR models employ "iterative refinement," passing initial guesses through multiple decoder layers to denoise the output. However, these models typically use a fixed number of iterations—a "naive solver" that may stop before reaching an optimal representation or exhibit instability.

Methodology: NAR as a Dynamical System

The core innovation of this paper is reimagining the NAR refinement process $z_t = f_\theta(z_{t-1}, x)$ as a dynamical system. Instead of stacking a predefined number of layers, DEQNAR uses a black-box root-finder (e.g., Anderson Acceleration) to directly find the equilibrium state $z^*$ where $z^* = f_\theta(z^*, x)$.

Overview of the DEQNAR framework compared to standard NAR, including convergence and BLEU score analysis.

Key Technical Components

  • Implicit Differentiation: Training utilizes the Implicit Function Theorem to back-propagate through the equilibrium point. This allows the model to simulate infinite depth with O(1) memory complexity during training.
  • Refinement Variants: The authors explore three modeling cases: Implicit (continuous hidden space), Explicit (discrete tokens with soft relaxation), and Mixed (hybrid approach combining hidden features with layer-wise prediction awareness).
  • Two-Stage Training: For explicit models, a denoising autoencoder pre-training phase is followed by DEQ fine-tuning to ensure robust convergence.

Experimental Results

The researchers evaluated DEQNAR on standard benchmarks including WMT14 (EN-DE), WMT16 (EN-RO), and IWSLT14 (DE-EN), using Knowledge Distillation from a powerful AR teacher.

Performance and Stability

DEQNAR consistently improved every NAR backbone tested (Vanilla NAT, GLAT, CTC, and VAE). A key finding is the direct correlation between convergence stability and translation quality. While standard NAR models often show "oscillations" or performance degradation with more iterations, DEQNAR remains stable.

BLEU score improvement and stability of DEQNAR (GLAT w/ DEQ) compared to the GLAT baseline as the number of function evaluations (NFE) increases.

Key Insights

  • Accuracy Gains: DEQNAR-IMPLICIT [CTC+VAE] achieved a competitive 28.10 BLEU on WMT14 EN-DE, outperforming many complex iterative models with fewer parameters.
  • Efficiency: Training a 6-layer GLAT model with DEQNAR required only 11.4GB of GPU memory, compared to 14.8GB for the baseline.
  • Flexibility: At inference time, users can trade off speed for quality by adjusting the convergence threshold of the root-finder.

Conclusion

DEQNAR successfully frames non-autoregressive sequence generation as an equilibrium-seeking process. By replacing fixed-layer stacks with implicit root-finding, the model achieves better stability and higher accuracy while maintaining memory efficiency. Future work may explore "KD-free" training and more native approaches to handling discrete state equilibrium.

Summary updated

Abstract

In this work, we argue that non-autoregressive (NAR) sequence generative models can equivalently be regarded as iterative refinement process towards the target sequence, implying an underlying dynamical system of NAR models: $ \mathbf{z} = \mathcal{f}(\mathbf{z}, \mathbf{x}) \rightarrow \mathbf{y}$. In such a way, the optimal prediction of a NAR model should be the equilibrium state of its dynamics if given infinitely many iterations. However, this is infeasible in practice due to limited computational and memory budgets. To this end, we propose DeqNAR to directly solve for the equilibrium state of NAR models based on deep equilibrium networks (Bai et al., 2019) with black-box root-finding solvers and back-propagate through the equilibrium point via implicit differentiation with constant memory. We conduct extensive experiments on four WMT machine translation benchmarks. Our main findings show that DeqNAR can indeed converge to a more accurate prediction and is a general-purpose framework that consistently yields substantial improvement for several strong NAR backbones.
BibTeX
@inproceedings{zheng2023deep,
  author = {Zaixiang Zheng and Yi Zhou and Hao Zhou},
  title = {{Deep Equilibrium Non-Autoregressive Sequence Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.747},
}
Figure