Title: Non-Semantics Suppressed Mask Learning for Unsupervised Video Semantic Compression
Introduction
Traditional video compression standards, such as HEVC and VVC, are optimized for human visual perception. They prioritize preserving textures and colors that the human eye finds pleasing while discarding data deemed imperceptible. However, as video consumption shifts from human viewers to AI systems—powering autonomous vehicles, security monitoring, and industrial automation—this paradigm becomes inefficient. AI models do not "see" pixels; they interpret semantics, such as object boundaries, motion trajectories, and structural relationships.
Existing Video Coding for Machine (VCM) methods often rely on hand-crafted features or require massive labeled datasets, limiting their scalability and task-agnosticism. This paper introduces the Semantic-Mining-then-Compensation (SMC) framework, an unsupervised approach to video semantic compression that optimizes data for AI "eyes" without requiring human-provided labels during training.
Methodology: The SMC Framework
The core of the proposed approach is the SMC framework, which operates as an enhancement layer over existing high-performance codecs like VVC.
The framework consists of several key components:
- Semantic Mining and Compensation: The system extracts semantic features from both the original high-quality video ($X$) and a compressed version ($\tilde{X}$). It identifies the "semantic residual"—the critical information lost during standard compression—and transmits only this additional data to the decoder.
- Unsupervised Masked Learning: To learn semantic representations without labels, the model utilizes Masked Image Modeling (MIM). By masking up to 90% of video patches and requiring the network to reconstruct the missing content, the model is forced to learn the underlying structural and temporal dynamics of the scene.
- Non-Semantics Suppressed (NSS) Learning: A significant challenge in generative learning is the tendency to reconstruct irrelevant visual noise. The authors introduce an NSS strategy using a Gaussian Mixture Model (GMM) to estimate the information entropy of video tokens. By explicitly penalizing the encoding of non-semantic information, the model focuses its limited bitrate on the most task-relevant features.
Experimental Setup
The model was trained on 60,000 unlabeled videos from the Kinetics400 dataset. Its performance was evaluated across three primary AI domains:
- Action Recognition: Tested on UCF101, HMDB51, Kinetics, and Diving48.
- Multiple Object Tracking (MOT): Evaluated using the MOT17 dataset.
- Video Object Segmentation (VOS): Assessed on DAVIS2017.
- Visual Quality: Benchmarked against HEVC Class C to ensure reasonable human-viewable quality.
Key Results and Discussion
The SMC framework demonstrates substantial improvements in both bitrate efficiency and task accuracy compared to traditional (HEVC, VVC) and learnable (FVC, PLVC) codecs.
As shown in the results, the method achieves a 31% to 48% BDBR saving on action recognition tasks compared to the VVC standard. In practical terms, this allows AI systems to maintain high accuracy while consuming nearly half the bandwidth. For instance, on the UCF101 dataset at a low bitrate of 0.04 bpp, the SMC framework improved recognition accuracy by 22% over leading learnable codecs like FVC.
Furthermore, the model's superiority in MOT and VOS tasks proves that the learned features are robust for both global motion analysis and local, pixel-precise segmentation. This task-agnostic nature is a direct result of the unsupervised semantic mining strategy, which captures fundamental video structures rather than optimizing for a single specific label.
Conclusion
The Semantic-Mining-then-Compensation framework represents a significant step toward efficient video compression for machine vision. By suppressing non-semantic information and focusing on structural residuals, the system achieves state-of-the-art performance in an unsupervised manner. While currently optimized for natural video domains, future work will explore adapting this approach to specialized fields like medical or industrial imaging and further optimizing decoder latency for real-time deployment.
Summary updated