Sparse Modular Activation for Efficient Sequence Modeling

Liliang Ren, Yang Liu, Shuohang Wang, Yichong Xu, Chenguang Zhu, ChengXiang Zhai

Abstract

Recent hybrid models combining Linear State Space Models (SSMs) with self-attention mechanisms have demonstrated impressive results across a range of sequence modeling tasks. However, current approaches apply attention modules statically and uniformly to all elements in the input sequences, leading to sub-optimal quality-efficiency trade-offs. To address this limitation, we introduce Sparse Modular Activation (SMA), a general mechanism enabling neural networks to sparsely and dynamically activate sub-modules for sequence elements in a differentiable manner. Through allowing each element to skip non-activated sub-modules, SMA reduces computation and memory consumption of neural networks at both training and inference stages. To validate the effectiveness of SMA on sequence modeling, we design a novel neural architecture, SeqBoat, which employs SMA to sparsely activate a Gated Attention Unit (GAU) based on the state representations learned from an SSM. By constraining the GAU to only conduct local attention on the activated inputs, SeqBoat can achieve linear inference complexity with theoretically infinite attention span, and provide substantially better quality-efficiency trade-off than the chunking-based models. With experiments on a wide range of tasks, including long sequence modeling, speech classification and language modeling, SeqBoat brings new state-of-the-art results among hybrid models with linear complexity, and reveals the amount of attention needed for each task through the learned sparse activation patterns.

1. Introduction

Modern sequence modeling involves a trade-off between the expressive power of Transformers and the efficiency of State Space Models (SSMs). While hybrid models attempt to bridge this gap, they often apply expensive attention modules uniformly across all input tokens. This static approach is computationally inefficient, as not all elements in a sequence require high-order modeling. For instance, common prepositions in text do not require the same associative recall power as rare technical nouns. We propose Sparse Modular Activation (SMA), a differentiable mechanism that enables networks to dynamically activate sub-modules only for relevant sequence elements.

2. Methods

The core of our approach is the SMA mechanism. It utilizes a "latent configurator"—a lightweight linear layer followed by a tempered softmax—to predict binary activation decisions for each token. To maintain efficiency on GPUs, we implement a parallel "compress-and-extract" strategy. This allows the model to process only the activated tokens through expensive sub-modules like the Gated Attention Unit (GAU).

The SeqBoat architecture integrates a linear SSM with a sparsely activated Gated Attention Unit (GAU) via the SMA mechanism.

We introduce SeqBoat, a novel architecture that combines a linear SSM with a sparsely activated GAU. By employing a Working Memory Mechanism, SeqBoat restricts the GAU to local attention on the compressed sequence, achieving linear inference complexity while maintaining a theoretically infinite attention span by focusing only on important historical tokens.

3. Experimental Results

We evaluated SeqBoat on the Long Range Arena (LRA) benchmark, speech classification, and language modeling.

3.1 Long Range Arena (LRA)

SeqBoat achieved an average accuracy of 87.62% on LRA, surpassing previous state-of-the-art hybrid models like MEGA-chunk. Notably, on the Text task (sequence length 4,096), SeqBoat demonstrated a 10.4x speedup and a 95% reduction in memory usage compared to standard Transformers.

Performance of SeqBoat compared to various quadratic, sub-quadratic, and linear complexity models on the LRA benchmark.

3.2 Insights from Activation Patterns

Analysis of learned activation patterns reveals that attention requirements are highly task-dependent. Image-based tasks (e.g., Pathfinder) require high activation across most layers, whereas text-based tasks utilize significantly less attention, particularly in early layers. This suggests that bottom layers specialize in first-order features via the SSM, while top layers invoke the GAU for complex second-order comparisons.

4. Conclusion

SeqBoat demonstrates that sparse, dynamic activation can significantly improve the efficiency of hybrid sequence models without sacrificing performance. While current implementations face some I/O bottlenecks during scatter operations, future optimizations with fused GPU kernels could further enhance speed. SMA offers a promising path for scaling long-context models efficiently to larger parameter counts and pre-training scenarios.

Summary updated

Abstract

Recent hybrid models combining Linear State Space Models (SSMs) with self-attention mechanisms have demonstrated impressive results across a range of sequence modeling tasks. However, current approaches apply attention modules statically and uniformly to all elements in the input sequences, leading to sub-optimal quality-efficiency trade-offs. To address this limitation, we introduce Sparse Modular Activation (SMA), a general mechanism enabling neural networks to sparsely and dynamically activate sub-modules for sequence elements in a differentiable manner. Through allowing each element to skip non-activated sub-modules, SMA reduces computation and memory consumption of neural networks at both training and inference stages. To validate the effectiveness of SMA on sequence modeling, we design a novel neural architecture, SeqBoat, which employs SMA to sparsely activate a Gated Attention Unit (GAU) based on the state representations learned from an SSM. By constraining the GAU to only conduct local attention on the activated inputs, SeqBoat can achieve linear inference complexity with theoretically infinite attention span, and provide substantially better quality-efficiency trade-off than the chunking-based models. With experiments on a wide range of tasks, including long sequence modeling, speech classification and language modeling, SeqBoat brings new state-of-the-art results among hybrid models with linear complexity, and reveals the amount of attention needed for each task through the learned sparse activation patterns. Our code is publicly available at https://github.com/renll/SeqBoat.
BibTeX
@inproceedings{ren2023sparse,
  author = {Liliang Ren and Yang Liu and Shuohang Wang and Yichong Xu and Chenguang Zhu and ChengXiang Zhai},
  title = {{Sparse Modular Activation for Efficient Sequence Modeling}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=TfbzX6I14i},
}
Figure