Abstract
Sentence encoders are fundamental components in Natural Language Processing, transforming variable-length text into fixed-size embeddings. Traditional self-attention mechanisms often rely on static weight vectors that apply the same internal logic regardless of the input sentence's unique structure. This paper introduces Dynamic Self-Attention (DSA), which utilizes an iterative dynamic routing algorithm—inspired by capsule networks—to adaptively compute word importance during inference. By combining a dense CNN backbone with this dynamic mechanism, the proposed model achieves state-of-the-art performance on the SNLI dataset while maintaining high parameter efficiency and training speed.
Introduction
A critical challenge in building sentence encoders is determining which words contribute most to the overall meaning. While standard self-attention has proven effective, its "one-size-fits-all" approach to weighting is sub-optimal for capturing the nuances of diverse linguistic contexts. The authors propose that the attention mechanism should be flexible enough to change its "point of view" depending on the specific sentence it processes. DSA addresses this by allowing the model to refine its focus through an iterative process, effectively "re-training" its attention for every new input.
Methodology
The DSA architecture consists of two primary stages: a feature extraction layer and the dynamic attention mechanism.
- Dense CNN Backbone: Instead of using sequential Recurrent Neural Networks (RNNs), the model employs a Convolutional Neural Network (CNN) with dense connections. This allows the model to capture local context efficiently and supports high levels of parallelization.
- Iterative Dynamic Routing: The core innovation is the adaptation of dynamic routing. Rather than a single-pass calculation, the model starts with an initial estimate of word importance and refines a "dynamic weight vector" ($z$) over several iterations (typically $r=2$). This vector evolves to represent the most informative parts of the specific sentence being processed.
- Multiple Attention Heads: To capture various linguistic features simultaneously (e.g., verbs vs. adjectives), the model can run multiple dynamic attention processes in parallel.
Experimental Results
The model was evaluated on the Stanford Natural Language Inference (SNLI) and Stanford Sentiment Treebank (SST) datasets.
- State-of-the-Art Performance: The "Multiple DSA" model achieved a test accuracy of 87.4% on SNLI, setting a new benchmark for sentence encoding methods at the time.
- Efficiency: A single-head DSA model reached 86.8% accuracy with only 2.1 million parameters, significantly fewer than competing models requiring 4 to 10 times the parameter count.
- Training Speed: Due to its CNN-based architecture, DSA trained in approximately 135–198 seconds per epoch, roughly 3 to 5 times faster than comparable high-performing RNN-based models.
Analysis and Visualization
One of the key strengths of DSA is its interpretability. By visualizing multiple attention heads, the authors observed emergent specialization. Different heads naturally learn to focus on distinct linguistic categories, such as actions, attributes, or specific entities.
Furthermore, Principal Component Analysis (PCA) of the dynamic weight vectors across 1,000 sentences showed a wide distribution, confirming that the model does not converge to a single global pattern but truly adapts its logic to the input sentence.
Conclusion and Future Work
Dynamic Self-Attention demonstrates that inference-time flexibility in attention mechanisms can lead to significant gains in both accuracy and efficiency. While the model showed the most dramatic improvements on large-scale datasets like SNLI, its performance on smaller datasets like SST suggests that the dynamic mechanism thrives on complex, high-volume data. Future research could explore the integration of dynamic routing into Transformer-based architectures, which have since become the standard in the field.
Summary updated