Fast Directional Self-Attention Mechanism

Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Chengqi ZhangarXiv 2018 · OpenReview

Abstract

Self-attention mechanisms have largely superseded recurrent and convolutional architectures in natural language processing due to their superior parallelizability and ability to model long-range dependencies. However, standard self-attention often forces a trade-off between computational efficiency and expressive power. This paper introduces Multi-mask Tensorized Self-Attention (MTSA), a novel framework that captures both pairwise (token-to-token) and global (source-to-token) dependencies without the memory overhead typically associated with high-dimensional alignment. By reformulating tensor operations into optimized matrix multiplications and utilizing a multi-head masking strategy for sequence order, MTSA achieves state-of-the-art performance across nine NLP benchmarks while remaining as fast and memory-efficient as CNN-based models.

Introduction

In the current NLP landscape, self-attention is the dominant paradigm. Despite its success, existing mechanisms frequently suffer from two primary limitations. First, they often fail to simultaneously model local pairwise interactions and global contextual importance effectively. Second, increasing the expressivity of these models—for instance, through multi-dimensional attention—usually leads to a "memory explosion" where computation time and memory usage grow quadratically with sequence length.

The challenge addressed in this work is the development of a self-attention mechanism that possesses the expressivity of multi-dimensional attention but operates with the efficiency of standard dot-product attention. MTSA is proposed as a solution that bridges this gap, offering a high-rank alignment representation through a computationally efficient implementation.

Methods

The MTSA architecture is built upon three core technical innovations designed to maximize both representational power and hardware efficiency.

Tensorized Self-Attention (TSA)

MTSA integrates two distinct attention functions: a scaled dot-product function to capture pairwise token-to-token dependencies and a multi-dimensional additive function to model global source-to-token dependencies. These are fused to generate feature-wise alignment scores, providing a much richer contextual representation than standard scalar-based attention.

Computation-Optimized Matrix Math

A significant hurdle for multi-dimensional attention is the requirement to store and process high-rank tensors (e.g., $n \times n \times d_h$). The authors introduce a mathematical reformulation that allows the same feature-wise attention output to be computed using standard 2D matrix multiplications. This optimization bypasses the memory bottleneck, allowing the model to scale linearly in terms of memory consumption relative to sequence length.

Multi-Masking for Sequence Order

To compensate for the inherent lack of word order in self-attention, MTSA employs a multi-head architecture where different heads are assigned specific positional masks. Forward masks hide future tokens, while backward masks hide past tokens. This approach injects bi-directional sequential information into the model without the need for computationally expensive recurrent layers.

The architecture of the Multi-mask Tensorized Self-Attention (MTSA) mechanism, illustrating the fusion of token-to-token and source-to-token dependencies along with the masking strategy.

Results

The authors evaluated MTSA across a diverse suite of nine NLP benchmarks, including Natural Language Inference (NLI), Semantic Role Labeling (SRL), and various sentence classification tasks.

Efficiency and Scalability

Empirical results demonstrate that MTSA is exceptionally fast, performing 4 to 5 times faster than RNN-based counterparts. As shown in the performance analysis, MTSA matches the speed and memory footprint of CNNs and standard dot-product multi-head attention, even as sequence lengths increase.

Performance metrics for MTSA compared to other architectures: (a) Feed-forward memory usage, (b) Feed-forward time cost, and (c) Inference time vs. memory consumption with accuracy labels.

Benchmark Performance

  • Natural Language Inference: On the SNLI dataset, MTSA achieved a state-of-the-art accuracy of 86.3% using only 2.9 million parameters. This outperforms more complex models, including those based on parsing trees and RNN-attention hybrids.
  • Sentence Classification: MTSA dominated across multiple classification tasks, achieving top results on CR (84.9%), MPQA (90.5%), and TREC (95.3%).
  • Machine Translation: Integrating MTSA into the encoder of a Transformer model led to a statistically significant improvement in BLEU scores on the WMT14 English-German task (27.21 vs. 26.98).

Conclusion

Multi-mask Tensorized Self-Attention (MTSA) represents a significant advancement in efficient self-attention design. By combining the expressive benefits of tensorized alignment with the speed of optimized matrix operations, it provides a powerful alternative to both standard Transformers and recurrent models. Future research directions include scaling MTSA for more complex generative tasks like text summarization and exploring advanced masking techniques to further enhance its ability to process long-form documents.

Summary updated

BibTeX
@misc{shen2018fast,
  author = {Tao Shen and Tianyi Zhou and Guodong Long and Jing Jiang and Chengqi Zhang},
  title = {{Fast Directional Self-Attention Mechanism}},
  year = {2018},
  url = {https://openreview.net/forum?id=5ZrMs6510gp},
}
Figure