Why and when should you pool? Analyzing Pooling in Recurrent Architectures

Pratyush Maini, Keshav Kolluru, Danish Pruthi, Mausam

Abstract

Pooling-based recurrent neural architectures consistently outperform their counterparts without pooling. However, the reasons for their enhanced performance are largely unexamined. In this work, we examine three commonly used pooling techniques (mean-pooling, max-pooling, and attention), and propose max-attention, a novel variant that effectively captures interactions among predictive tokens in a sentence. We find that pooling-based architectures substantially differ from their non-pooling equivalents in their learning ability and positional biases--which elucidate their performance benefits. By analyzing the gradient propagation, we discover that pooling facilitates better gradient flow compared to BiLSTMs. Further, we expose how BiLSTMs are positionally biased towards tokens in the beginning and the end of a sequence. Pooling alleviates such biases. Consequently, we identify settings where pooling offers large benefits: (i) in low resource scenarios, and (ii) when important words lie towards the middle of the sentence. Among the pooling techniques studied, max-attention is the most effective, resulting in significant performance gains on several text classification tasks.

Introduction

In Natural Language Processing (NLP), Bidirectional LSTMs (BiLSTMs) have long served as a foundational architecture for sequence classification. It is common practice to append a "pooling" layer—such as mean-pooling, max-pooling, or attention—to aggregate hidden states into a single summary vector for classification. While empirical evidence consistently shows that pooling improves performance, the underlying mechanics have remained poorly understood. This research investigates whether pooling is merely a superior information compression technique or if it fundamentally alters the model's learning dynamics and positional biases.

Methods

The study compares standard BiLSTMs against three common pooling variants: Mean-pooling, Max-pooling, and Attention.

Architectural comparison of standard pooling (left) and attention-based (right) mechanisms.

A key contribution of this work is the introduction of Max-attention (MAXATT). Unlike standard attention, which typically employs a fixed, learnable query vector for the entire dataset, MAXATT utilizes the max-pooled representation of the specific input sentence as the query. This approach allows the model to use the most salient features already identified to dynamically determine which other parts of the sentence require focus.

To evaluate these models, the researchers utilized two diagnostic metrics:

  1. Vanishing Ratio: Measures gradient strength in the middle of a sequence relative to its ends.
  2. Normalized Word Importance (NWI): Quantifies the drop in model confidence when a specific word is replaced by an UNK token, revealing which parts of the sentence the model prioritizes.

Key Insights and Ideas

The Gradient "Shortcut"

In a standard BiLSTM, gradients must backpropagate through every time step to reach the center of a sequence. During the early stages of training, these gradients often vanish before reaching middle tokens. Pooling layers provide a "direct flight" for gradients, allowing them to reach any hidden state directly from the output.

Gradient norm across word positions, highlighting the severe vanishing gradient problem in standard BiLSTMs compared to pooling methods.

As shown in the figure above, the gradient norm in the middle of a sequence for standard BiLSTMs can be $10^{-15}$ times smaller than at the ends. Pooling layers effectively eliminate this disparity, ensuring uniform gradient flow across the entire sequence and accelerating learning, particularly in low-resource settings.

Eliminating Positional Bias

BiLSTMs without pooling exhibit "tunnel vision," showing a heavy bias toward tokens at the beginning and end of a sequence. This makes them fragile when critical information is located in the middle of a sentence. Pooling acts as a "great equalizer," making the model position-invariant and allowing it to identify salient tokens regardless of their location.

Key Results

  • Low-Resource Performance: In scenarios with only 1,000 training examples, pooling methods outperformed standard BiLSTMs by 13-14% on the IMDb dataset.
  • Robustness to Noise: The researchers tested models by injecting random Wikipedia sentences into various parts of the input. While standard BiLSTMs' accuracy plummeted when noise was added to the middle, pooling-based models (especially MAXATT) remained resilient.
  • Effectiveness of MAXATT: The proposed Max-attention method outperformed other pooling techniques in 80% of the tested cases, demonstrating its ability to capture complex inter-word dependencies through sentence-specific queries.

Conclusion

This work demonstrates that pooling is essential for recurrent architectures not just for information aggregation, but for maintaining healthy gradient flow and mitigating positional biases. While Transformers inherently possess global connectivity that avoids these specific issues, these insights provide a theoretical basis for the continued use and refinement of pooling in recurrent models.

Summary updated

Abstract

Pooling-based recurrent neural architectures consistently outperform their counterparts without pooling. However, the reasons for their enhanced performance are largely unexamined. In this work, we examine three commonly used pooling techniques (mean-pooling, max-pooling, and attention), and propose max-attention, a novel variant that effectively captures interactions among predictive tokens in a sentence. We find that pooling-based architectures substantially differ from their non-pooling equivalents in their learning ability and positional biases--which elucidate their performance benefits. By analyzing the gradient propagation, we discover that pooling facilitates better gradient flow compared to BiLSTMs. Further, we expose how BiLSTMs are positionally biased towards tokens in the beginning and the end of a sequence. Pooling alleviates such biases. Consequently, we identify settings where pooling offers large benefits: (i) in low resource scenarios, and (ii) when important words lie towards the middle of the sentence. Among the pooling techniques studied, max-attention is the most effective, resulting in significant performance gains on several text classification tasks.
BibTeX
@inproceedings{maini2020why,
  author = {Pratyush Maini and Keshav Kolluru and Danish Pruthi and Mausam},
  title = {{Why and when should you pool? Analyzing Pooling in Recurrent Architectures}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=ao6Kp9GL_l0},
}
Figure