Why and when should you pool? Analyzing Pooling in Recurrent Architectures

Pratyush Maini, Keshav Kolluru, Danish Pruthi, Mausam

Abstract

Pooling-based recurrent neural architectures consistently outperform their counterparts without pooling on sequence classification tasks. However, the reasons for their enhanced performance are largely unexamined. In this work, we examine three commonly used pooling techniques (mean-pooling, max-pooling, and attention), and propose max-attention, a novel variant that captures interactions among predictive tokens in a sentence. Using novel experiments, we demonstrate that pooling architectures substantially differ from their non-pooling equivalents in their learning ability and positional biases: (i) pooling facilitates better gradient flow than BiLSTMs in initial training epochs, and (ii) BiLSTMs are biased towards tokens at the beginning and end of the input, whereas pooling alleviates this bias. Consequently, we find that pooling yields large gains in low resource scenarios, and instances when salient words lie towards the middle of the input. Across several text classification tasks, we find max-attention to frequently outperform other pooling techniques.

Introduction

In Natural Language Processing (NLP), Recurrent Neural Networks (RNNs) like BiLSTMs have long served as fundamental building blocks for sequence modeling. To convert a sequence of hidden states into a fixed-length sentence representation for classification, practitioners often employ pooling—taking the maximum, average, or a weighted "attention" sum. While pooling is a standard "trick" that empirically improves performance, the underlying mechanisms for its success remained poorly understood. This paper investigates whether BiLSTMs suffer from a "blind spot" in the middle of sentences and how pooling mechanisms mitigate these issues through improved gradient flow and reduced positional bias.

Methodology: From Standard Pooling to Max-attention

The authors evaluate three standard pooling techniques: Mean-pooling, Max-pooling, and Attention. In a standard BiLSTM, the final hidden state is typically used as the sequence representation. In contrast, pooling mechanisms aggregate information across all time steps.

Architectural comparison between standard pooling (left) and attention-based pooling (right).

A key contribution of this work is the introduction of Max-attention (MAXATT). While standard attention uses a fixed, learnable query vector across the entire dataset, MAXATT uses the max-pooled representation of the specific input sentence as the query. This allows the model to identify the most salient features in a sentence first and then use those features to dynamically attend to supporting context elsewhere in the sequence.

Analyzing Gradient Flow and Positional Bias

The study identifies two primary reasons for the superiority of pooling:

  1. The Gradient Highway: In standard BiLSTMs, gradients for tokens in the middle of a sequence vanish significantly during the early stages of training, sometimes reaching scales as low as $10^{-15}$. This creates a learning delay where the model focuses on the ends of the sentence first. Pooling provides a direct mathematical path from the loss function to every hidden state, acting as a "gradient highway" that enables learning from the entire sentence from the first batch.
  2. Positional Invariance: BiLSTMs exhibit a deep-seated bias toward the beginning and end of inputs. By using a metric called Normalized Word Importance (NWI)—which measures model confidence drops when words are masked—the authors show that pooling-based models are more "position-blind," focusing on salient keywords regardless of their location.
Gradient norms across word positions (a) and training dynamics (b, c) showing how pooling (MAXATT) maintains better gradient flow and achieves faster convergence than standard BiLSTMs.

Experimental Results

The models were tested on four major benchmarks: IMDb, Yahoo! Answers, Amazon Reviews, and Yelp Reviews, with a focus on sequences longer than 100 words.

  • Low-Resource Gains: The benefits of pooling are most pronounced in data-scarce environments. With only 1,000 training samples, pooling architectures outperformed standard BiLSTMs by 13-14% in accuracy.
  • MAXATT Performance: The proposed Max-attention method was the top performer in approximately 80% of experimental cases, demonstrating that local sentence context is a more effective query for attention than a global learnable vector.
  • Robustness to Noise: When sequences were padded with random "noise" text, pooling models (especially Max-pooling and Attention) remained resilient, whereas BiLSTM performance plummeted due to its reliance on positional cues.

Conclusion

This research reveals that pooling is more than a simple compression technique; it is a structural fix for the inherent limitations of recurrent architectures. By facilitating immediate gradient flow to the middle of sequences and eliminating positional bias, pooling enables models to be more data-efficient and robust. While these insights are specific to RNNs—as Transformers inherently address these issues via self-attention—they provide a rigorous foundation for choosing pooling strategies in recurrent sequence modeling.

Summary updated

Abstract

Pooling-based recurrent neural architectures consistently outperform their counterparts without pooling on sequence classification tasks. However, the reasons for their enhanced performance are largely unexamined. In this work, we examine three commonly used pooling techniques (mean-pooling, max-pooling, and attention, and propose *max-attention*, a novel variant that captures interactions among predictive tokens in a sentence. Using novel experiments, we demonstrate that pooling architectures substantially differ from their non-pooling equivalents in their learning ability and positional biases: (i) pooling facilitates better gradient flow than BiLSTMs in initial training epochs, and (ii) BiLSTMs are biased towards tokens at the beginning and end of the input, whereas pooling alleviates this bias. Consequently, we find that pooling yields large gains in low resource scenarios, and instances when salient words lie towards the middle of the input. Across several text classification tasks, we find max-attention to frequently outperform other pooling techniques.
BibTeX
@inproceedings{maini2020why,
  author = {Pratyush Maini and Keshav Kolluru and Danish Pruthi and Mausam},
  title = {{Why and when should you pool? Analyzing Pooling in Recurrent Architectures}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=ao6Kp9GL_l0},
}
Figure