RingFormer: Rethinking Recurrent Transformer with Adaptive Level Signals

Jaemu Heo, Eldor Fozilov, Hyunmin Song, Taehwan KimEMNLP 2025

TL;DR

This paper introduces RingFormer, a novel recurrent Transformer architecture that significantly reduces parameter count by reusing a single Transformer block iteratively. It achieves this by incorporating input-dependent "level signals" generated via low-rank matrices, allowing the model to adapt its behavior at each recurrent step while maintaining high performance on translation and image classification tasks.

What Problem Does This Paper Address?

Transformers have revolutionized sequential data processing, but their success comes at a high computational and memory cost due to their large number of parameters. While parameter sharing and recurrence have been explored to mitigate this, they often lead to a performance drop compared to the original Transformer. This paper addresses the challenge of creating a parameter-efficient Transformer that can match the performance of its larger counterparts, particularly in resource-constrained environments.

Methods

The core innovation of RingFormer is its recurrent architecture, which reuses a single Transformer block multiple times. To enable this block to adapt to different "levels" (iterations), the authors introduce adaptive level signals. These signals are generated using low-rank matrices, inspired by Parameter-Efficient Fine-Tuning (PEFT) techniques like LoRA. Specifically, the input is projected to a lower-dimensional space and then back, creating a subtle, input-dependent adjustment. These level signals are applied separately to the attention and feedforward sub-layers within the Transformer block. The paper also notes that they use unique layer normalization for each level to further aid adaptation.

Datasets

The paper evaluates RingFormer on two standard benchmarks:

  • WMT-14 German-English translation: A widely used dataset for machine translation.
  • ImageNet-1K (and a subset called ImageNet-small): Used for image classification tasks.

Key Insights and Ideas

  • Recurrence with Adaptation is Key: Simply reusing Transformer layers (like in Universal Transformers) isn't enough. The ability to adapt the reused layer's behavior based on the input at each step is crucial for maintaining performance.
  • Low-Rank Signals for Efficiency: Using low-rank matrices to generate adaptive level signals is a clever way to introduce input-dependent adaptation without significantly increasing the parameter count. This is a key enabler of RingFormer's efficiency.
  • Mimicking Original Transformer Behavior: The paper's analysis (CKA and MAD) suggests that RingFormer's layer representations and attention patterns more closely resemble the original Transformer than other recurrent models, explaining its competitive performance.
  • Level Signals as "Nudges": The authors hypothesize that these signals act as gentle nudges to guide the recurrent layers, rather than drastically altering the input, which helps preserve the learning process.

Key Results

  • Translation: RingFormer achieves competitive BLEU scores on WMT-14 German-English translation, outperforming Universal Transformer with similar or fewer parameters. For instance, a base-size RingFormer model (8.94M parameters) achieved 29.52 BLEU, comparable to Universal Transformer (7.34M parameters, 29.12 BLEU) and One Wide FFN (20.98M parameters, 29.54 BLEU), while significantly outperforming them in terms of parameter efficiency.
  • Image Classification: On ImageNet-small, RingFormer achieved the best performance among models with comparable parameter counts, and when scaled up, it matched or exceeded baseline recurrent models. For example, on ImageNet-small, a RingFormer model (4.4M parameters) achieved 60.66% accuracy, outperforming UiT (3.60M parameters, 58.64%) and One Wide FFN (4.51M parameters, 58.62%).
  • Parameter Efficiency: RingFormer consistently demonstrates a significant reduction in parameter count compared to standard Transformers and even other recurrent models, while maintaining high performance.

Limitations and Future Work

  • Computational Overhead: RingFormer introduces additional computations for generating level signals, though the authors argue this is efficient due to the low-rank nature of the signals.
  • Limited Scope of Experiments: Due to computational constraints, the authors could not conduct experiments on large-scale language modeling tasks.
  • Future Work: The authors suggest exploring RingFormer's effectiveness on billion-parameter models and in domains like language modeling. They also note that increasing the rank of the low-rank decomposition can further improve performance, suggesting a trade-off between parameter count and accuracy that can be tuned.

Summary updated

BibTeX
@inproceedings{heo2025ringformer,
  author = {Jaemu Heo and Eldor Fozilov and Hyunmin Song and Taehwan Kim},
  title = {{RingFormer: Rethinking Recurrent Transformer with Adaptive Level Signals}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure