Introduction
Modern Transformer models are computationally expensive because they apply a uniform amount of computation to every input token, regardless of its complexity. While a patch of blue sky requires minimal processing, it currently consumes the same FLOPs as a complex object. Existing efficiency techniques like token dropping or early exiting primarily adjust the depth of the model. Adaptive Computation Modules (ACMs) introduce a more granular approach by dynamically adjusting the width—the representational capacity within a layer—on a per-token basis.
Methodology: The Adaptive Computation Module
The ACM architecture replaces standard large weight matrices with a sequence of $N$ smaller neural modules, termed "learners." The final output of an ACM is the cumulative sum of the outputs from the first $k$ learners, where $k$ is determined dynamically for each token.
A lightweight, trainable gating network analyzes each input token to decide the optimal number of learners ($k$) to execute. To maintain differentiability during training, the authors employ the Gumbel-Softmax trick. The transition from a static pre-trained model to an ACM-enabled one follows a three-stage "ACMization" pipeline:
- Module Distillation: Learners are trained to approximate the output of a specific layer in the original model using Mean Squared Error (MSE).
- Gating Pre-training: The gates are trained to predict the minimum $k$ required to meet specific accuracy thresholds.
- End-to-End Fine-tuning: The entire system is optimized with auxiliary losses to prevent gating collapse and ensure computational efficiency.
To translate these algorithmic gains into real-world performance, the authors developed custom GPU kernels using Triton, allowing for parallel execution and efficient token reordering on modern hardware.
Experimental Evaluation
The researchers evaluated ACMs across two distinct domains: Computer Vision (ImageNet-1k with ViT-B) and Speech-to-Text (CommonVoice with XLS-R-300M).
Key Insights: Semantic Awareness and Granularity
Analysis of the gating decisions reveals that width-based computation is highly granular. In most scenarios, the full capacity of a layer is unnecessary; a majority of tokens can be processed using only one or two learners.
Furthermore, the model exhibits emergent semantic awareness. As shown in Figure 1, the gating mechanism automatically allocates more computation to semantically rich areas—such as the features of a bird or a ladybug—while reducing resources for background elements. This behavior is achieved without explicit spatial or semantic supervision, demonstrating that the model learns to prioritize informative features to maintain accuracy under a computational budget.
Results and Performance
ACMs demonstrate Pareto dominance over existing baselines like A-ViT and MoEfication. On ImageNet-1k, the approach provides a superior trade-off between accuracy and FLOPs, particularly at lower computational budgets. In speech tasks, ACMs consistently achieved lower Word Error Rates (WER) than MoEfication across all evaluated budgets. Crucially, the custom Triton implementation ensures that reductions in average learner count translate to near-linear improvements in wall-clock latency on A100 GPUs.
Conclusion and Future Work
Adaptive Computation Modules provide a robust framework for per-token width scaling in Transformers. By replacing monolithic layers with a sequence of learners and a smart gate, the authors achieve significant efficiency gains without sacrificing performance. While currently focused on MLP blocks and linear projections, future work could extend this logic to the self-attention mechanism or combine ACMs with other techniques like quantization to further push the boundaries of inference efficiency.
Summary updated