Abstract
Despite the great progress in video understanding made by deep convolutional neural networks, feature representation learned by existing methods may be biased to static visual cues. To address this issue, we propose a novel method to suppress static visual cues (SSVC) based on probabilistic analysis for self-supervised video representation learning. In our method, video frames are first encoded to obtain latent variables under standard normal distribution via normalizing flows. By modelling static factors in a video as a random variable, the conditional distribution of each latent variable becomes shifted and scaled normal. Then, the less-varying latent variables along time are selected as static cues and suppressed to generate motion-preserved videos. Finally, positive pairs are constructed by motion-preserved videos for contrastive learning to alleviate the problem of representation bias to static cues. The less-biased video representation can be better generalized to various downstream tasks.
Introduction: The Challenge of Representation Bias
In video recognition, deep learning models often rely on "shortcuts"—identifying actions based on static backgrounds (e.g., seeing water and guessing "swimming") rather than actual movement. This representation bias hinders a model's ability to perform true temporal reasoning. If a model focuses on the "where" (static cues) instead of the "how" (temporal dynamics), it fails to generalize to new environments or distinguish between similar actions in different settings. The challenge lies in forcing the model to ignore the scene and focus on the action itself.
Methodology: The S2VC Framework
The proposed S2VC (Suppressing Static Visual Cues) framework addresses this by isolating and suppressing static information within a contrastive learning pipeline. The core innovation involves a multi-step process to manipulate video data:
- Latent Encoding with Normalizing Flows (NF): Unlike standard networks, NF uses invertible functions to map video frames into a latent space where data follows a standard normal distribution. This allows for precise, mathematically grounded manipulation of features.
- Static Cue Identification: By modeling static factors as random variables, the researchers identify latent variables that vary minimally over time as "static cues" (e.g., background or scene elements).
- Suppression and Reconstruction: These static variables are zeroed out to minimize their information entropy. Because NF is invertible, the modified latent vector can be transformed back into pixel space, creating a motion-preserved video where the background is effectively removed while the motion remains intact.
- Contrastive Learning: The model is trained to recognize the original video and its motion-preserved counterpart as "positive pairs," forcing the 3D CNN backbone to extract features invariant to the background.
Key Insights and Temporal Reasoning
A major advantage of Normalizing Flows is their mathematical invertibility, allowing for "surgical" removal of specific features in the latent space. To verify that the model learns motion rather than static shortcuts, it was tested on a "temporal-order" task: distinguishing natural videos from shuffled ones.
The S2VC-trained model significantly outperformed supervised baselines in this task, proving it had a much better grasp of temporal dynamics and was less likely to rely on static scene information.
Experimental Results
The researchers evaluated S2VC on standard benchmarks including UCF101, HMDB51, and Kinetics-400. Key findings include:
- Action Recognition: S2VC achieved a 5.2% improvement in Top-1 accuracy on UCF101 and an 8.6% improvement on HMDB51 compared to the MoCo baseline.
- Video Retrieval: The learned features proved more discriminative, consistently outperforming state-of-the-art methods that rely solely on the RGB modality.
- State-of-the-Art: The method outperformed existing techniques for pre-training when only single RGB modality is used.
Conclusion and Limitations
S2VC effectively mitigates representation bias by suppressing static visual cues through probabilistic latent analysis. While the use of Normalizing Flows introduces some computational overhead—mitigated in this study by spatial down-sampling—the resulting motion-centric representations offer superior performance in action recognition and temporal reasoning. Future work may explore integrating S2VC with multi-modal approaches, such as optical flow or audio, to further enhance video understanding.
This summary is based on the research paper "Suppressing Static Visual Cues via Normalizing Flows for Self-Supervised Video Representation Learning."
Summary updated