Adaptive Computation Modules: Granular Conditional Computation For Efficient Inference

Bartosz Wójcik, Alessio Devoto, Karol Pustelnik, Pasquale Minervini, Simone Scardapane

Abstract

The computational cost of transformer models makes them inefficient in low-latency or low-power applications. While techniques such as quantization or linear attention can reduce the computational load, they may incur a reduction in accuracy. In addition, globally reducing the cost for all inputs may be sub-optimal. We observe that for each layer, the full width of the layer may be needed only for a small subset of tokens inside a batch and that the "effective" width needed to process a token can vary from layer to layer. Motivated by this observation, we introduce the Adaptive Computation Module (ACM), a generic module that dynamically adapts its computational load to match the estimated difficulty of the input on a per-token basis. An ACM consists of a sequence of learners that progressively refine the output of their preceding counterparts. An additional gating mechanism determines the optimal number of learners to execute for each token. We also describe a distillation technique to replace any pre-trained model with an "ACMized" variant. The distillation phase is designed to be highly parallelizable across layers while being simple to plug-and-play into existing networks. Our evaluation of transformer models in computer vision and speech recognition demonstrates that substituting layers with ACMs significantly reduces inference costs without degrading the downstream accuracy for a wide interval of user-defined budgets.

Introduction

Transformer models have become the backbone of modern AI, but their static nature leads to significant computational waste. Currently, every token—whether it represents a detailed object in the foreground or a blurry patch of background sky—is processed using the exact same amount of mathematical operations. This "one-size-fits-all" approach is inefficient for low-latency or low-power applications.

While previous methods like token dropping or early exiting provide some relief, they are often too coarse. This paper introduces Adaptive Computation Modules (ACMs), a granular approach that scales computation up or down on a per-token basis within a single layer, designed specifically to be efficient on modern GPU hardware.

The Adaptive Computation Module (ACM)

Instead of a single large weight matrix, an ACM decomposes a layer into two main components:

  1. A Sequence of Learners: A set of $N$ small, homogeneous neural modules (e.g., two-layer MLPs). The final output of the ACM is the cumulative sum of the first $k$ learners.
  2. A Gating Network: A tiny, trainable network that analyzes the input token and predicts the optimal number of learners ($k$) required to process it.

The "ACMization" Process

To integrate ACMs into existing high-performance models, a three-phase training strategy is employed:

  • Phase 1: Module-wise Distillation: Individual layers in a pre-trained model are replaced with ACMs. The learners are trained to mimic the original layer's output using Mean Squared Error (MSE).
  • Phase 2: Gating Pre-training: The gating network is trained using labels generated by observing the minimum number of learners needed to reach a specific error threshold.
  • Phase 3: End-to-End Fine-tuning: The entire model is fine-tuned with auxiliary losses (entropy and diversity) to ensure the gating mechanism makes diverse choices and adheres to a user-defined computational budget ($\beta_{target}$).

Experimental Evaluation

The researchers evaluated ACMs across two distinct domains:

  • Computer Vision: On ImageNet-1k, ACMized Vision Transformers (ViT-B) achieved a superior Pareto frontier compared to competitors like A-ViT and MoEfication, maintaining high accuracy even with significant FLOP reductions.
  • Speech-to-Text: Using the CommonVoice-es dataset with a pre-trained XLS-R-300M model, ACMs achieved lower Word Error Rates (WER) across all evaluated computational budgets compared to existing baselines.

Key Insights and Semantic Awareness

Analysis of the gating mechanism reveals that ACMs naturally develop "semantic awareness." Without explicit supervision, the gate learns to allocate more computational resources to content-rich areas of an input while saving energy on redundant or "boring" regions.

Visualization of per-token computation allocation in a Vision Transformer. The heatmap shows that the ACM gating mechanism naturally allocates more learners (brighter colors) to the bird and fewer to the background sky.

Furthermore, the implementation of ACMs using custom Triton GPU kernels ensures that theoretical FLOP savings translate into real-world wall-clock speedups. The researchers demonstrated that latency on an NVIDIA A100 GPU scales linearly with the average number of learners used.

Conclusion and Future Work

ACMs provide a practical, plug-and-play solution for making Transformers more efficient through granular conditional computation. While the current work focuses on replacing MLP blocks and MHA projections, future research could extend this adaptive logic to the self-attention mechanism itself or combine it with quantization techniques to further optimize inference for edge devices.

Summary updated

Abstract

The computational cost of transformer models makes them inefficient in low-latency or low-power applications. While techniques such as quantization or linear attention can reduce the computational load, they may incur a reduction in accuracy. In addition, globally reducing the cost for all inputs may be sub-optimal. We observe that for each layer, the full width of the layer may be needed only for a small subset of tokens inside a batch and that the "effective" width needed to process a token can vary from layer to layer. Motivated by this observation, we introduce the Adaptive Computation Module (ACM), a generic module that dynamically adapts its computational load to match the estimated difficulty of the input on a per-token basis. An ACM consists of a sequence of learners that progressively refine the output of their preceding counterparts. An additional gating mechanism determines the optimal number of learners to execute for each token. We also describe a distillation technique to replace any pre-trained model with an "ACMized" variant. The distillation phase is designed to be highly parallelizable across layers while being simple to plug-and-play into existing networks. Our evaluation of transformer models in computer vision and speech recognition demonstrates that substituting layers with ACMs significantly reduces inference costs without degrading the downstream accuracy for a wide interval of user-defined budgets.
BibTeX
@inproceedings{wojcik2025adaptive,
  author = {Bartosz Wójcik and Alessio Devoto and Karol Pustelnik and Pasquale Minervini and Simone Scardapane},
  title = {{Adaptive Computation Modules: Granular Conditional Computation For Efficient Inference}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i20.35453},
}
Figure