Adaptive Computation Modules: Granular Conditional Computation for Efficient Inference

Bartosz Wójcik, Alessio Devoto, Karol Pustelnik, Pasquale Minervini, Simone Scardapane

Introduction

Modern Transformer models are computationally expensive because they apply a uniform amount of computation to every input token, regardless of its complexity. While a patch of blue sky requires minimal processing, it currently consumes the same FLOPs as a complex object. Existing efficiency techniques like token dropping or early exiting primarily adjust the depth of the model. Adaptive Computation Modules (ACMs) introduce a more granular approach by dynamically adjusting the width—the representational capacity within a layer—on a per-token basis.

Methodology: The Adaptive Computation Module

The ACM architecture replaces standard large weight matrices with a sequence of $N$ smaller neural modules, termed "learners." The final output of an ACM is the cumulative sum of the outputs from the first $k$ learners, where $k$ is determined dynamically for each token.

The ACM architecture uses a gating mechanism $g$ to select a subset of learners for each token.

A lightweight, trainable gating network analyzes each input token to decide the optimal number of learners ($k$) to execute. To maintain differentiability during training, the authors employ the Gumbel-Softmax trick. The transition from a static pre-trained model to an ACM-enabled one follows a three-stage "ACMization" pipeline:

  1. Module Distillation: Learners are trained to approximate the output of a specific layer in the original model using Mean Squared Error (MSE).
  2. Gating Pre-training: The gates are trained to predict the minimum $k$ required to meet specific accuracy thresholds.
  3. End-to-End Fine-tuning: The entire system is optimized with auxiliary losses to prevent gating collapse and ensure computational efficiency.

To translate these algorithmic gains into real-world performance, the authors developed custom GPU kernels using Triton, allowing for parallel execution and efficient token reordering on modern hardware.

Experimental Evaluation

The researchers evaluated ACMs across two distinct domains: Computer Vision (ImageNet-1k with ViT-B) and Speech-to-Text (CommonVoice with XLS-R-300M).

Key Insights: Semantic Awareness and Granularity

Analysis of the gating decisions reveals that width-based computation is highly granular. In most scenarios, the full capacity of a layer is unnecessary; a majority of tokens can be processed using only one or two learners.

Visualization of computation allocation: ACMs naturally assign more learners to semantically important image regions.

Furthermore, the model exhibits emergent semantic awareness. As shown in Figure 1, the gating mechanism automatically allocates more computation to semantically rich areas—such as the features of a bird or a ladybug—while reducing resources for background elements. This behavior is achieved without explicit spatial or semantic supervision, demonstrating that the model learns to prioritize informative features to maintain accuracy under a computational budget.

Results and Performance

ACMs demonstrate Pareto dominance over existing baselines like A-ViT and MoEfication. On ImageNet-1k, the approach provides a superior trade-off between accuracy and FLOPs, particularly at lower computational budgets. In speech tasks, ACMs consistently achieved lower Word Error Rates (WER) than MoEfication across all evaluated budgets. Crucially, the custom Triton implementation ensures that reductions in average learner count translate to near-linear improvements in wall-clock latency on A100 GPUs.

Conclusion and Future Work

Adaptive Computation Modules provide a robust framework for per-token width scaling in Transformers. By replacing monolithic layers with a sequence of learners and a smart gate, the authors achieve significant efficiency gains without sacrificing performance. While currently focused on MLP blocks and linear projections, future work could extend this logic to the self-attention mechanism or combine ACMs with other techniques like quantization to further push the boundaries of inference efficiency.

Summary updated

Abstract

The computational cost of transformer models makes them inefficient in low-latency or low-power applications. While techniques such as quantization or linear attention can reduce the computational load, they may incur a reduction in accuracy. In addition, globally reducing the cost for all inputs may be sub-optimal. We observe that for each layer, the full width of the layer may be needed only for a small subset of tokens inside a batch and that the "effective" width needed to process a token can vary from layer to layer. Motivated by this observation, we introduce the Adaptive Computation Module (ACM), a generic module that dynamically adapts its computational load to match the estimated difficulty of the input on a per-token basis. An ACM consists of a sequence of learners that progressively refine the output of their preceding counterparts. An additional gating mechanism determines the optimal number of learners to execute for each token. We also describe a distillation technique to replace any pre-trained model with an "ACMized" variant. The distillation phase is designed to be highly parallelizable across layers while being simple to plug-and-play into existing networks. Our evaluation of transformer models in computer vision and speech recognition demonstrates that substituting layers with ACMs significantly reduces inference costs without degrading the downstream accuracy for a wide interval of user-defined budgets.
BibTeX
@inproceedings{wojcik2025adaptive,
  author = {Bartosz Wójcik and Alessio Devoto and Karol Pustelnik and Pasquale Minervini and Simone Scardapane},
  title = {{Adaptive Computation Modules: Granular Conditional Computation for Efficient Inference}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i20.35453},
}
Figure