TL;DR
DiSAN (Directional Self-Attention Network) is a lightweight neural network for sentence encoding that completely removes the need for recurrent (RNN) or convolutional (CNN) layers. By introducing multi-dimensional and directional self-attention, it achieves state-of-the-art accuracy across multiple NLP tasks while being significantly faster and more parameter-efficient than traditional models.
What Problem Does This Paper Address?
For years, the NLP community has relied on Recurrent Neural Networks (RNNs) to capture long-range dependencies and Convolutional Neural Networks (CNNs) to extract local features. However, both have significant drawbacks: RNNs are inherently sequential, making them difficult to parallelize and slow to train, while CNNs struggle with global context.
While attention mechanisms have emerged as a powerful alternative, they often lose temporal order information—the sequence in which words appear—which is crucial for understanding language. The authors of DiSAN set out to build a model that retains the efficiency and flexibility of attention while explicitly modeling the directional and feature-level nuances of human language.
Methods
The core of the paper lies in two novel attention mechanisms that form the building blocks of DiSAN:
- Multi-dimensional Attention: Traditional attention computes a single scalar score for each word. DiSAN instead computes a feature-wise score vector. Since words are represented by high-dimensional embeddings, this allows the model to attend to specific features of a word that are relevant to the current context, effectively handling polysemy (words with multiple meanings). Traditional vs. Multi-dimensional Attention
- Directional Self-Attention (DiSA): To capture temporal order without using RNNs, the authors use positional masks. By applying forward and backward masks to the self-attention matrix, the model can explicitly learn asymmetric dependencies (e.g., how a word relates only to those that came before it). The Directional Self-Attention (DiSA) Mechanism
The full DiSAN architecture works in three main stages:
- Context Fusion: The input sequence is processed by parallel forward and backward DiSA blocks to capture context from both directions.
- Fusion Gate: A gate mechanism combines the original input with the attention output, similar to the gates found in LSTMs but applied to attention.
- Sequence Compression: A final "source2token" multi-dimensional attention block compresses the entire sequence into a single vector representation for downstream tasks.
Datasets
The authors evaluated DiSAN on a comprehensive suite of eight benchmark datasets:
- Natural Language Inference: SNLI (Stanford Natural Language Inference) and MultiNLI (Multi-Genre NLI).
- Sentiment Analysis: Stanford Sentiment Treebank (SST).
- Semantic Relatedness: SICK (Sentences Involving Compositional Knowledge).
- Sentence Classification: Customer Review (CR), MPQA (opinion polarity), SUBJ (subjectivity), and TREC (question-type classification).
Key Insights and Ideas
- Attention is all you need (for more than just NMT): While the "Transformer" model showed attention's power for translation, DiSAN proves it can be the sole engine for general-purpose sentence encoding across a wide variety of tasks.
- Feature-Level Selection: By making attention multi-dimensional, the model can "pick and choose" which parts of a word's embedding are important. A heatmap analysis in the paper shows that the model generates very different attention vectors for the word "glass" when it refers to a bottle versus a drink.
- Order via Masking: You don't need sequential processing to understand order. Simple matrix masks (forward, backward, diagonal) can inject structural priors into a parallelizable attention mechanism.
- Efficiency Gains: Because it relies on matrix multiplications rather than sequential steps, DiSAN is roughly 3x faster to train than a standard Bi-LSTM.
Key Results
- New SOTA on SNLI: DiSAN achieved a test accuracy of 85.62%, outperforming all other sentence-encoding models at the time, including those using complex tree structures or much larger parameter counts.
- Sentiment Analysis: On the SST dataset, it reached 51.72% accuracy for fine-grained classification, surpassing previous CNN and RNN-based leaders.
- Parameter Efficiency: DiSAN achieved these results with only 2.35M parameters, making it smaller than many competing models like the 300D NSE (3.0M) or 600D Bi-LSTM (2.8M).
Limitations and Future Work
The authors noted that while DiSAN excels on standard sentence lengths, its performance begins to degrade on very long sequences (e.g., movie reviews longer than 38 words). This suggests that while self-attention captures long-range dependencies well, there may still be a "dilution" effect when normalizing over very long sequences.
Future work suggested by the authors includes applying these directional attention mechanisms to more complex, multi-step reasoning tasks like Question Answering and Machine Reading Comprehension.
Summary updated