Abstract
This paper introduces a generalized pooling mechanism for neural networks, specifically focusing on vector-based multi-head attention. By allowing multiple attention "heads" to assign vector-specific weights to hidden states, the model captures complex semantic nuances more effectively than traditional max, mean, or scalar-attention pooling. The authors demonstrate state-of-the-art results across natural language inference, sentiment analysis, and author profiling tasks.
Introduction
In natural language processing, models like Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs) generate sequences of hidden states. However, downstream tasks such as classification require a single, fixed-size "sentence embedding." The standard approach to bridge this gap is pooling. Common methods include max pooling and mean pooling, which are computationally efficient but context-blind. While recent scalar self-attention mechanisms improved this by weighting words, they apply a single weight to the entire word vector. This paper argues that different dimensions of a word's representation may carry different semantic importance, necessitating a more granular, vector-based approach.
Methodology
The proposed Generalized Pooling framework introduces three key innovations:
- Vector-based Attention: Instead of a single scalar weight per word, the model computes a weight vector. This allows for an element-wise (Hadamard) product with hidden states, enabling the model to selectively emphasize specific features (dimensions) of a word's representation.
- Multi-head Mechanism: To capture diverse linguistic features, the model employs multiple attention heads. Each head can focus on different aspects of the sentence (e.g., predicates vs. modifiers), with the final embedding being a concatenation of these multi-head outputs.
- Redundancy Penalization: To prevent multiple heads from converging on the same information, the authors introduce penalization terms to the loss function. These terms encourage diversity across parameter matrices, attention matrices, or the resulting embeddings.
Experimental Results
The model was evaluated on four benchmarks: SNLI, MultiNLI, the Yelp sentiment dataset, and an Age profiling dataset. As shown in the results for the Stanford Natural Language Inference (SNLI) task, the generalized pooling approach significantly outperforms previous strong baselines.
On MultiNLI, the model achieved 73.8% (In-domain) and 74.0% (Cross-domain) accuracy. It also showed substantial gains in sentiment analysis (Yelp: 66.55%) and author profiling (Age: 82.63%), proving the robustness of the architecture across different NLP domains.
Analysis and Discussion
A critical finding of the study is the consistent superiority of vector-based attention over scalar-based attention. As training progresses, vector-based models maintain a performance lead, particularly as the number of attention heads increases.
The authors highlight that vector attention is a mathematical generalization of existing techniques; by adjusting weights, it can emulate max or mean pooling. Furthermore, the ablation study emphasized that redundancy penalization is essential; without it, the multi-head mechanism loses its effectiveness as heads become redundant.
Conclusion
Generalized pooling via vector-based multi-head attention provides a powerful and flexible way to generate sentence embeddings. By moving beyond the "blunt instrument" of scalar weights and enforcing diversity through penalization, this framework sets a new standard for sentence representation in deep learning models.
Summary updated