Dynamic Self-Attention : Computing Attention over Words Dynamically for Sentence Embedding

Deunsol Yoon, Dongbok Lee, SangKeun LeearXiv 2018 · arXiv

Abstract

Sentence encoders are fundamental components in Natural Language Processing, transforming variable-length text into fixed-size embeddings. Traditional self-attention mechanisms often rely on static weight vectors that apply the same internal logic regardless of the input sentence's unique structure. This paper introduces Dynamic Self-Attention (DSA), which utilizes an iterative dynamic routing algorithm—inspired by capsule networks—to adaptively compute word importance during inference. By combining a dense CNN backbone with this dynamic mechanism, the proposed model achieves state-of-the-art performance on the SNLI dataset while maintaining high parameter efficiency and training speed.

Introduction

A critical challenge in building sentence encoders is determining which words contribute most to the overall meaning. While standard self-attention has proven effective, its "one-size-fits-all" approach to weighting is sub-optimal for capturing the nuances of diverse linguistic contexts. The authors propose that the attention mechanism should be flexible enough to change its "point of view" depending on the specific sentence it processes. DSA addresses this by allowing the model to refine its focus through an iterative process, effectively "re-training" its attention for every new input.

Methodology

The DSA architecture consists of two primary stages: a feature extraction layer and the dynamic attention mechanism.

  1. Dense CNN Backbone: Instead of using sequential Recurrent Neural Networks (RNNs), the model employs a Convolutional Neural Network (CNN) with dense connections. This allows the model to capture local context efficiently and supports high levels of parallelization.
  2. Iterative Dynamic Routing: The core innovation is the adaptation of dynamic routing. Rather than a single-pass calculation, the model starts with an initial estimate of word importance and refines a "dynamic weight vector" ($z$) over several iterations (typically $r=2$). This vector evolves to represent the most informative parts of the specific sentence being processed.
  3. Multiple Attention Heads: To capture various linguistic features simultaneously (e.g., verbs vs. adjectives), the model can run multiple dynamic attention processes in parallel.
The architecture of the Dynamic Self-Attention model, showing the flow from input sentence to the final sentence embedding via a dense CNN and dynamic routing.

Experimental Results

The model was evaluated on the Stanford Natural Language Inference (SNLI) and Stanford Sentiment Treebank (SST) datasets.

  • State-of-the-Art Performance: The "Multiple DSA" model achieved a test accuracy of 87.4% on SNLI, setting a new benchmark for sentence encoding methods at the time.
  • Efficiency: A single-head DSA model reached 86.8% accuracy with only 2.1 million parameters, significantly fewer than competing models requiring 4 to 10 times the parameter count.
  • Training Speed: Due to its CNN-based architecture, DSA trained in approximately 135–198 seconds per epoch, roughly 3 to 5 times faster than comparable high-performing RNN-based models.

Analysis and Visualization

One of the key strengths of DSA is its interpretability. By visualizing multiple attention heads, the authors observed emergent specialization. Different heads naturally learn to focus on distinct linguistic categories, such as actions, attributes, or specific entities.

Visualization of four different attention heads across two sentences, demonstrating how the model specializes its focus on different linguistic features.

Furthermore, Principal Component Analysis (PCA) of the dynamic weight vectors across 1,000 sentences showed a wide distribution, confirming that the model does not converge to a single global pattern but truly adapts its logic to the input sentence.

Conclusion and Future Work

Dynamic Self-Attention demonstrates that inference-time flexibility in attention mechanisms can lead to significant gains in both accuracy and efficiency. While the model showed the most dramatic improvements on large-scale datasets like SNLI, its performance on smaller datasets like SST suggests that the dynamic mechanism thrives on complex, high-volume data. Future research could explore the integration of dynamic routing into Transformer-based architectures, which have since become the standard in the field.

Summary updated

BibTeX
@misc{yoon2018dynamic,
  author = {Deunsol Yoon and Dongbok Lee and SangKeun Lee},
  title = {{Dynamic Self-Attention : Computing Attention over Words Dynamically for Sentence Embedding}},
  year = {2018},
  eprint = {1808.07383},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1808.07383},
}
Figure