Enhancing Sentence Embedding with Generalized Pooling

Qian Chen, Zhen-Hua Ling, Xiaodan Zhu

Abstract

This paper introduces a generalized pooling mechanism for neural networks, specifically focusing on vector-based multi-head attention. By allowing multiple attention "heads" to assign vector-specific weights to hidden states, the model captures complex semantic nuances more effectively than traditional max, mean, or scalar-attention pooling. The authors demonstrate state-of-the-art results across natural language inference, sentiment analysis, and author profiling tasks.

Introduction

In natural language processing, models like Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs) generate sequences of hidden states. However, downstream tasks such as classification require a single, fixed-size "sentence embedding." The standard approach to bridge this gap is pooling. Common methods include max pooling and mean pooling, which are computationally efficient but context-blind. While recent scalar self-attention mechanisms improved this by weighting words, they apply a single weight to the entire word vector. This paper argues that different dimensions of a word's representation may carry different semantic importance, necessitating a more granular, vector-based approach.

Methodology

The proposed Generalized Pooling framework introduces three key innovations:

  1. Vector-based Attention: Instead of a single scalar weight per word, the model computes a weight vector. This allows for an element-wise (Hadamard) product with hidden states, enabling the model to selectively emphasize specific features (dimensions) of a word's representation.
  2. Multi-head Mechanism: To capture diverse linguistic features, the model employs multiple attention heads. Each head can focus on different aspects of the sentence (e.g., predicates vs. modifiers), with the final embedding being a concatenation of these multi-head outputs.
  3. Redundancy Penalization: To prevent multiple heads from converging on the same information, the authors introduce penalization terms to the loss function. These terms encourage diversity across parameter matrices, attention matrices, or the resulting embeddings.

Experimental Results

The model was evaluated on four benchmarks: SNLI, MultiNLI, the Yelp sentiment dataset, and an Age profiling dataset. As shown in the results for the Stanford Natural Language Inference (SNLI) task, the generalized pooling approach significantly outperforms previous strong baselines.

Table 1: Test accuracy on the SNLI dataset. The proposed 1200D BiLSTM with generalized pooling achieves state-of-the-art performance (86.6%) among sentence-encoding models.

On MultiNLI, the model achieved 73.8% (In-domain) and 74.0% (Cross-domain) accuracy. It also showed substantial gains in sentiment analysis (Yelp: 66.55%) and author profiling (Age: 82.63%), proving the robustness of the architecture across different NLP domains.

Analysis and Discussion

A critical finding of the study is the consistent superiority of vector-based attention over scalar-based attention. As training progresses, vector-based models maintain a performance lead, particularly as the number of attention heads increases.

Figure 1: Comparison of development set accuracy on SNLI between vector-based and scalar-based attention mechanisms across different numbers of heads (1, 3, 5, 7, 9).

The authors highlight that vector attention is a mathematical generalization of existing techniques; by adjusting weights, it can emulate max or mean pooling. Furthermore, the ablation study emphasized that redundancy penalization is essential; without it, the multi-head mechanism loses its effectiveness as heads become redundant.

Conclusion

Generalized pooling via vector-based multi-head attention provides a powerful and flexible way to generate sentence embeddings. By moving beyond the "blunt instrument" of scalar weights and enforcing diversity through penalization, this framework sets a new standard for sentence representation in deep learning models.

Summary updated

BibTeX
@inproceedings{chen2018enhancing,
  author = {Qian Chen and Zhen-Hua Ling and Xiaodan Zhu},
  title = {{Enhancing Sentence Embedding with Generalized Pooling}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=gu5uELsc6E},
}
Figure