For years, video understanding models have operated under a major compromise. Processing raw video pixels is incredibly expensive; a single video contains thousands of frames, making backpropagation through deep 3D convolutional networks or video transformers a memory and computational nightmare.
To bypass this bottleneck, the traditional paradigm has relied on decoupled feature extraction. In this setup, researchers pass raw video through a frozen, pre-trained backbone (such as an I3D or ResNet model trained on ImageNet or Kinetics classification), save the resulting feature embeddings to disk, and train a lightweight temporal model on top of those static features.
While computationally efficient, this decoupled approach introduces a fundamental performance bottleneck: the end-to-end optimization gap.
The Frozen Backbone Bottleneck
When you freeze a visual backbone, you create an insurmountable barrier for gradient descent. The downstream temporal model—whether it is localizing an action boundary, answering a question, or grounding a text query—cannot pass gradients back to reshape the underlying visual representations. The gradient flow hits a brick wall at the feature boundary.
[Raw Video] ──> [ Frozen Backbone ] ──> [ Static Features ] ──x──> [ Downstream Head ]
(Gradients Stop Here)
This isolation hurts performance because general-purpose backbones are rarely optimized for the specific nuances of downstream tasks. As Chen-Lin Zhang et al. note in their work on ActionFormer, relying on frozen features from a fixed backbone network severely limits a model's ability to learn task-specific representations that capture fine-grained action boundaries.
The core issue is a mismatch in granularity:
- Classification vs. Localization: A backbone pre-trained on video classification is optimized to identify what is in the video globally (e.g., "swimming"). However, tasks like Temporal Action Localization (TAL) require the feature space to be highly sensitive to exact start and end coordinates. As Jiyang Gao et al. point out, a frozen feature extractor cannot adapt its low-level spatial or temporal filters to detect precise boundary transitions.
- Cross-Modal Alignment: In video-language tasks like video grounding, the visual features must align with complex textual queries. Kevin Qinghong Lin et al. observe that frozen backbones prevent the model from optimizing for task-specific temporal features, missing out on the tight semantic alignment that comes from joint video-text training.
Without end-to-end integration, downstream models are forced to work with "hand-crafted" or legacy features that cannot adapt, leaving significant performance on the table (Runhao Zeng et al.).
Why End-to-End Learning Works
Transitioning to an end-to-end architecture allows the downstream task's loss function to directly reshape how the backbone perceives raw pixels. If a model needs to distinguish between "opening a door" and "closing a door," end-to-end training allows the early layers of the network to tune their filters to the subtle directional motions required for that specific distinction, rather than relying on generic appearance features.
[Raw Video] ──> [ Trainable Backbone ] ──> [ Dynamic Features ] ──> [ Downstream Head ]
│ │
└─────────────────── Learning Gradients ─────────────┘
To make this joint optimization computationally viable, researchers have developed two primary strategies: sparse sampling and unified architectures.
1. Sparse Sampling
Instead of forcing the GPU to process every single frame of a long video during backpropagation, models can sample the video intelligently.
In ClipBERT, Jie Lei et al. introduce a "less is more" philosophy for video-and-language learning. By sparsely sampling just one or a few short clips during each training step, they reduce the memory footprint enough to make end-to-end backpropagation affordable. Because the visual backbone is updated at every step, it quickly adapts to the specific nuances of the video-text relationship, easily outperforming models trained on dense but frozen features.
2. Streamlined, Unified Pipelines
Another approach replaces multi-stage pipelines filled with hand-crafted rules and pre-extracted expert features with single-shot, unified networks.
For instance, Yao Hu proposes TadTR, an end-to-end Transformer that treats temporal action detection as a direct set prediction problem. By unifying the pipeline, the detection head directly influences representation learning, eliminating the need for decoupled feature extraction stages.
Similarly, in the realm of online action detection, E2E-LOAD by Shuqiang Cao et al. breaks the reliance on frozen backbones by pairing a specialized "Space-then-Space-time" architecture with a token-reuse inference mechanism. This allows the model to achieve state-of-the-art accuracy through end-to-end fine-tuning while maintaining high computational efficiency.
Conclusion
The shift away from decoupled feature extraction toward end-to-end representation learning represents a major evolution in video understanding. By finding clever ways to open up the frozen backbone bottleneck—whether through sparse sampling or unified architectures—modern models allow downstream task demands to flow all the way back to raw pixels. The result is a visual representation explicitly tuned for temporal boundaries, motion nuances, and cross-modal alignment.