Computer vision models designed for video understanding frequently inherit a design pattern from static image classification: they compress spatial dimensions into a single, global feature vector before performing temporal reasoning. While this global average pooling or holistic frame tokenization works well for broad scene classification, it creates a severe information bottleneck for complex video tasks.
When a model collapses a frame's spatial dimensions, it discards the precise "where and when" of the scene. This loss of spatial granularity makes it exceptionally difficult for models to track small objects, distinguish subtle human-object interactions, or isolate localized movements from dominant background noise. To overcome this limitation, a growing body of research focuses on architectures that preserve spatial granularity throughout the video-modeling pipeline.
The Global Pooling Bottleneck
In standard video analysis pipelines, frames are passed through deep convolutional networks or vision transformers to extract high-level semantic features. At the end of this spatial extraction phase, global average pooling is typically applied to reduce the data footprint.
As Brais Martínez et al. demonstrate in Action recognition with spatial-temporal discriminative filter banks, this aggregation washes out the fine-grained details required to separate highly similar activities, such as "eating a burger" versus "eating a hot dog." Because the global vector summarizes the entire frame, subtle motion signatures and small, critical objects are obscured.
This over-compression introduces several distinct failure modes across video understanding tasks:
- Background Overfitting: When spatial features are averaged globally, large, static background regions contribute disproportionately to the final vector. Kun-Yu Lin et al. show in Diversifying Spatial-Temporal Perception for Video Domain Generalization that this allows models to become "lazy," relying on easy, domain-specific background cues (like the specific walls of a gym) rather than the actual action (like a basketball shot).
- Loss of Temporal Directionality: Summarizing frames into global tokens limits a model's ability to detect localized spatial changes over time. For example, Piyush Nitin Bagad et al. demonstrate in Chirality in Action: Time-Aware Video Representation Learning by Latent Straightening that standard video models struggle to differentiate between forward and time-reversed actions, such as "opening a door" versus "closing a door," because they lack dense, region-specific tracking.
- Signal Dilution: In specialized domains like surgical skill assessment or e-commerce video captioning, the primary action often involves tiny tools or specific product features. As Zhenqiang Li et al. note in Surgical Skill Assessment via Video Semantic Aggregation, global pooling creates a bottleneck that prevents the model from separating these critical small objects from the surrounding background tissue or environment.
Architectures for Retaining Spatial Detail
To bypass the global pooling bottleneck, researchers have developed alternative representation paradigms that maintain spatial structure across time. These approaches broadly fall into three categories: dense latent layouts, entity-centric representations, and localized attention graphs.
Dense Latent Layouts
Instead of collapsing a frame into a vector, some frameworks enforce a dense spatial layout within the latent space itself. Single-author work by Tengda Han, Video Representation Learning by Dense Predictive Coding, demonstrates that self-supervised models can learn high-level action semantics without manual labels by predicting the future latent states of video blocks while explicitly preserving their dense spatial coordinates.
Standard Pipeline:
[Frame] ──> [Spatial CNN] ──> [Global Pooling] ──> [1D Vector] ──> [Temporal Model]
(Spatial Structure Lost)
Dense/Topology-Preserving Pipelines:
[Frame] ──> [Spatial CNN] ──> [Dense Latent Map / Convolutions] ──> [Temporal Model]
(Spatial Structure Retained)
Similarly, Nicolas Ballas et al. introduced the GRU-RCN architecture in Delving Deeper into Convolutional Networks for Learning Video Representations Nicolas Ballas, which replaces the fully connected layers of standard recurrent units with convolutions. By doing so, the network preserves spatial topology across multiple resolutions, allowing it to capture fine-grained pixel-level motion trajectories over time.
Entity-Centric Tracking
Another prominent strategy is to decompose a video frame into distinct semantic "entities" or objects rather than rigid spatial grids. Matthew Walmer et al. propose this shift in Multi-entity Video Transformers for Fine-Grained Video Representation Learning. Using a strategy called Learnable Spatial Token Pooling (LSTP), their model represents each frame as a collection of distinct entities (such as actors, tools, and backgrounds) and tracks them across time, avoiding the "one frame, one vector" bottleneck entirely.
This object-aware approach is highly effective for language-generation tasks like video captioning. In Object-Aware Aggregation With Bidirectional Temporal Graph for Video Captioning, Junchao Zhang et al. bypass simple temporal averaging by building a bidirectional temporal graph that tracks specific objects. This ensures that the language generation component retains access to the fine-grained visual details necessary to describe specific entities involved in an action.
Localized Attention and Graphs
Transformers offer powerful mechanisms for long-range modeling, but global self-attention across an entire video volume is computationally prohibitive and often dilutes local context. To address this, Chao Qin et al. developed a Spatial-Temporal Deformable Attention (STDA) mechanism in A Spatial-Temporal Deformable Attention based Framework for Breast Lesion Detection in Videos. Instead of aggregating information globally, STDA samples features locally across consecutive frames, providing the precise spatial specificity required to track small or blurry objects.
When long-range modeling is required, videos can be represented as sparse semantic graphs rather than dense volumes. Yang Wang et al. demonstrate in Supervoxel Attention Graphs for Long-Range Video Modeling that compressing a video into a sparse graph of "supervoxel" nodes preserves object-level structure and spatial relationships over long durations, achieving high accuracy with a fraction of the memory required by standard 3D CNNs.
Why Preserving Granularity Works: The Intuition
The core reason why preserving spatial granularity works can be understood through the lens of relational reasoning. When a model collapses a frame into a single vector, the temporal aggregator can only reason about the video as a sequence of holistic scenes:
$$\text{Scene}_1 \rightarrow \text{Scene}_2 \rightarrow \text{Scene}_3$$
If the background remains constant, the difference between $\text{Scene}_1$ and $\text{Scene}_2$ in the latent space may be negligible, causing the model to miss the action entirely or overfit to static environmental features.
By contrast, preserving spatial granularity—whether through dense feature maps, object tokens, or localized graphs—allows the temporal aggregator to perform relational tracking over time:
$$\text{Entity A (Hand)} \times \text{Entity B (Door Handle)} \times \text{Spatial Coordinates } (x, y, z)$$
This structural preservation changes how the network processes motion. Instead of attempting to decode subtle movements from a blurred global mixture, the model tracks how specific regions or entities shift relative to one another.
As shown by Jinglin Xu et al. in FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessment, breaking down a complex action into localized, sequential steps prevents the model from taking shortcuts. It forces the architecture to evaluate the precise nuances of body position and rotation speed against an exemplar, rather than relying on global scene features that look identical across different athletes.
Ultimately, preserving spatial granularity ensures that the temporal modeling component receives structured, high-fidelity inputs. By protecting the "where" within the visual representation, these architectures unlock a significantly deeper and more robust understanding of the "what" and "when."
Key Foundational Work
Motion Segmentation for Time-Varying Mesh Sequences Based on Spherical Registration (2009) — contributed 1 concept elements to this direction.
Nanjing University in TRECVID 2010 (2010) — contributed 1 concept elements to this direction.