TL;DR
MemCAM is a novel transformer-based framework that successfully trains temporal cross-attention for video segmentation using only video-level labels, eliminating the need for noisy external priors like optical flow. By propagating Class Activation Maps (CAMs) through feature-space similarity, it creates a robust self-supervisory signal that stabilizes training and significantly improves temporal consistency and segmentation accuracy.
What Problem Does This Paper Address?
Video Semantic Segmentation (VSS) is essential for applications ranging from autonomous driving to industrial automation. While foundation models like SAM 2 have set new benchmarks, they require massive amounts of pixel-level annotations—a luxury not available in specialized domains like industrial waste sorting or defect detection.
Weakly Supervised VSS (WS-VSS) attempts to bridge this gap by learning from video-level labels (e.g., "this video contains a car"). However, training the temporal "cross-attention" weights that link frames together is notoriously unstable in label-scarce settings. Without dense supervision, these weights often fail to converge, sometimes even performing worse than models that look at each frame in isolation. To compensate, existing methods often lean on external priors like optical flow or color consistency. Unfortunately, these cues are non-semantic, computationally expensive, and fragile under changing lighting or occlusions.
Methods
The authors introduce MemCAM, which replaces external priors with internal feature-space self-supervision. The architecture (shown in Figure 2) consists of three primary novel components:
- Memory Attention Module (MAM): A lightweight transformer block that allows the model to refine the current frame's features by querying a "Memory Bank" of embeddings from the $T$ most recent frames.
- CAM Propagation Algorithm (CAMPA): This is the core "teacher" mechanism. Instead of supervising attention weights directly, CAMPA synthesizes a "synthetic CAM" for the current frame. It does this by matching patches in the current frame to similar patches in past frames within the latent feature space and "dragging" their activation scores forward.
- CAM Consistency Loss: This loss function forces the model's predicted CAM to align with the synthetic one generated by CAMPA. This stabilizes the learning of temporal relationships by providing a consistent, semantic target.
The CAMPA process (detailed in Figure 3) effectively "recycles" reliable activations from the past to guide the present, ensuring that the model maintains a coherent view of objects as they move through time.
Datasets
The model was evaluated on the only two publicly available benchmarks that match the video-level supervision protocol:
- YouTube-Objects v2.2: A general-domain dataset used to evaluate temporal object localization (CorLoc).
- $WS^2$: A domain-specific industrial dataset for waste sorting, used to evaluate segmentation performance (mIoU).
Key Insights and Ideas
- Decoupling Context from Supervision: The authors found that you shouldn't supervise cross-attention weights directly in weak settings. Instead, supervise their effect at the CAM level through feature-space propagation.
- Feature Similarity as a Semantic Prior: Unlike optical flow, which just tracks pixels, feature-space matching in a pre-trained transformer captures semantic relationships. This makes the supervisory signal much cleaner and more robust to noise.
- The "Tiny" Advantage: Interestingly, the "Tiny" variant of MemCAM often outperformed the "Small" variant on the $WS^2$ dataset. This suggests a capacity-data mismatch: larger models can overfit on the limited diversity of small, weakly labeled datasets, whereas smaller models generalize better.
- Localization vs. Classification: The research confirms that higher classification accuracy doesn't necessarily mean better localization. In fact, MemCAM's localization improved even as its classification accuracy slightly drifted, showing that the consistency loss successfully reshapes spatial activations independently of the class label.
Key Results
MemCAM sets a new state-of-the-art for WS-VSS across both benchmarks:
- YouTube-Objects v2.2: MemCAM-tiny achieved 76.8% CorLoc, an 8.2% improvement over the single-frame baseline (MCTformer+).
- $WS^2$ Dataset: MemCAM-tiny reached 41.21% mIoU, outperforming both image-based methods and temporal methods using external cues like optical flow (POF-CAM at 38.70%).
Qualitatively, MemCAM produces masks that are significantly more complete and less "flickery" than baseline methods, as seen in Figure 1.
The ablation study (summarized in the table below) highlights that the combination of the Memory Bank (M) and the CAM Consistency Loss ($L_{CAM}$) is crucial; using memory without the specific loss actually degrades performance.
Limitations and Future Work
- Teacher Dependency: The quality of the synthetic targets is upper-bounded by the initial quality of the "teacher" (the frozen backbone). If the backbone produces very poor initial CAMs, propagation cannot fully fix them.
- Abrupt Changes: The feature-space affinities can degrade during very fast motion, heavy occlusions, or shot cuts, where the visual appearance changes too drastically for the memory bank to provide useful context.
- Memory Horizon: The current FIFO (First-In-First-Out) memory queue is short (typically 5 frames). Objects that leave the scene and return much later are not "remembered."
Future work aims to extend this self-supervisory mechanism to even more challenging regimes, such as fully unsupervised or semi-supervised video semantic segmentation.
Summary updated