Global Temporal Pooling for Sequence Representation

314 concepts257 papers1 proposal

Deep learning models for video, audio, and text often face a structural mismatch: while the input is a variable-length sequence of frames or tokens, the final decision—such as "what action is this?" or "is this audio a deepfake?"—requires a single, fixed-length representation. Temporal feature aggregation (or temporal pooling) is the mechanism used to collapse these sequences into a unified global signature.

The Consensus of the Bag of Frames

The simplest way to understand a video is to treat it as a "bag of frames." Instead of modeling complex chronological dependencies, researchers often extract features from individual frames and aggregate them using statistical operations.

Venugopalan et al. demonstrate that temporal mean pooling—averaging feature vectors across the time dimension—provides a robust summary of visual content. This works because, for many tasks like video captioning, the overall presence of objects and actions is more important than their exact millisecond-by-millisecond order. Similarly, Feng et al. show that simply averaging frame-level features from a pre-trained ImageNet model is remarkably effective for multimedia event detection, capturing the general visual context without the overhead of recurrent architectures.

Salience and Extremes

While mean pooling captures the "average" state, it can wash out brief, critical events. To capture these, researchers use max pooling or min/max aggregation.

  • Identifying Salience: In Visual Question Answering, Ji and Wang use global max-pooling to identify the most relevant features across a text sequence, making the model invariant to where a specific keyword appears.
  • Capturing Motion: For action recognition, Chéron et al. employ min/max aggregation on body part tracks. By recording the extreme values of features over time, the model captures the range of a movement—the highest point of a jump or the furthest reach of a limb—which defines the action better than an average pose would.

Exploiting Spatio-Temporal Persistence

Temporal aggregation works because of a property called persistence: objects in a video do not change their identity or appearance instantly from one frame to the next.

Giunchiglia et al. argue that averaging embeddings across a sequence exploits this persistence to smooth out noise and transient occlusions. This creates a more stable "identity signature" than any single frame could provide. This intuition is echoed in face verification; Sokolova and Kharchevnikova found that calculating the average feature vector of all frames in a face track significantly outperforms selecting a single "best" frame, as the aggregate captures the subject across multiple angles and lighting conditions.

Efficiency and Dimensionality Reduction

Beyond improving accuracy, aggregation is a vital tool for computational efficiency. High-resolution temporal data is expensive to store and search.

In the context of audio deepfake detection, Kang et al. use time-wise feature aggregation to condense high-resolution embeddings. This reduces the storage and latency costs of comparing a suspect clip against a massive database of authentic audio. Similarly, Lei demonstrates that placing a global average pooling layer between temporal modeling and classification allows deep layers to process summarized "segment-level" information rather than raw frame-level scores, enabling state-of-the-art audio tagging with 30 times fewer parameters.

Beyond Simple Statistics

As tasks become more complex, simple averaging may lose too much information. Researchers have developed more sophisticated ways to "distill" sequences:

  • Trainable Pooling: Shin et al. utilize architectures like NetVLAD, which learn to cluster frame-level descriptors into a fixed-size representation. This captures temporal dynamics more effectively than simple averaging by learning which visual patterns are most representative of a category.
  • Hierarchical Distillation: For long videos, Li et al. use specialized tokens to aggregate frame-level features into "shot embeddings." This hierarchical approach allows the model to reason about the relationship between different scenes without being overwhelmed by the total number of individual frames.
  • Global Context: Jonghwan Mun shows that for short clips, a simple global context embedding—compressing the entire video volume through fully connected layers—can compete with complex attention mechanisms by capturing the overall transition of states.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 247 papers
  2. Version 1 · 2026-10-04 · 276 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_temporal_feature_aggregation_temporal_fe_49550229,
  title = {temporal feature aggregation (temporal feature pooling)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-feature-aggregation-temporal-feature-pooling/rdr_49550229bb9c4ce4a41daf6181a0234f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_temporal_feature_aggregation_temporal_fe_49550229_v2,
  title = {temporal feature aggregation (temporal feature pooling)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-feature-aggregation-temporal-feature-pooling/rdr_49550229bb9c4ce4a41daf6181a0234f/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake