Unlocking Cross-Attention in Label-Scarce Regimes: A Weakly Supervised Approach for Video Semantic Segmentation

Andrea Marelli, Alessandro Ardenghi, Giacomo Boracchi, Luca MagriarXiv 2026 · arXiv

TL;DR

MemCAM is a novel transformer-based framework that successfully trains temporal cross-attention for video segmentation using only video-level labels, eliminating the need for noisy external priors like optical flow. By propagating Class Activation Maps (CAMs) through feature-space similarity, it creates a robust self-supervisory signal that stabilizes training and significantly improves temporal consistency and segmentation accuracy.

What Problem Does This Paper Address?

Video Semantic Segmentation (VSS) is essential for applications ranging from autonomous driving to industrial automation. While foundation models like SAM 2 have set new benchmarks, they require massive amounts of pixel-level annotations—a luxury not available in specialized domains like industrial waste sorting or defect detection.

Weakly Supervised VSS (WS-VSS) attempts to bridge this gap by learning from video-level labels (e.g., "this video contains a car"). However, training the temporal "cross-attention" weights that link frames together is notoriously unstable in label-scarce settings. Without dense supervision, these weights often fail to converge, sometimes even performing worse than models that look at each frame in isolation. To compensate, existing methods often lean on external priors like optical flow or color consistency. Unfortunately, these cues are non-semantic, computationally expensive, and fragile under changing lighting or occlusions.

Methods

The authors introduce MemCAM, which replaces external priors with internal feature-space self-supervision. The architecture (shown in Figure 2) consists of three primary novel components:

  1. Memory Attention Module (MAM): A lightweight transformer block that allows the model to refine the current frame's features by querying a "Memory Bank" of embeddings from the $T$ most recent frames.
  2. CAM Propagation Algorithm (CAMPA): This is the core "teacher" mechanism. Instead of supervising attention weights directly, CAMPA synthesizes a "synthetic CAM" for the current frame. It does this by matching patches in the current frame to similar patches in past frames within the latent feature space and "dragging" their activation scores forward.
  3. CAM Consistency Loss: This loss function forces the model's predicted CAM to align with the synthetic one generated by CAMPA. This stabilizes the learning of temporal relationships by providing a consistent, semantic target.
Overview of the MemCAM architecture showing the flow from inputs to the Memory Bank and the CAMPA-driven consistency loss.

The CAMPA process (detailed in Figure 3) effectively "recycles" reliable activations from the past to guide the present, ensuring that the model maintains a coherent view of objects as they move through time.

The CAMPA algorithm propagates past activations based on patch-level feature similarity.

Datasets

The model was evaluated on the only two publicly available benchmarks that match the video-level supervision protocol:

  • YouTube-Objects v2.2: A general-domain dataset used to evaluate temporal object localization (CorLoc).
  • $WS^2$: A domain-specific industrial dataset for waste sorting, used to evaluate segmentation performance (mIoU).

Key Insights and Ideas

  • Decoupling Context from Supervision: The authors found that you shouldn't supervise cross-attention weights directly in weak settings. Instead, supervise their effect at the CAM level through feature-space propagation.
  • Feature Similarity as a Semantic Prior: Unlike optical flow, which just tracks pixels, feature-space matching in a pre-trained transformer captures semantic relationships. This makes the supervisory signal much cleaner and more robust to noise.
  • The "Tiny" Advantage: Interestingly, the "Tiny" variant of MemCAM often outperformed the "Small" variant on the $WS^2$ dataset. This suggests a capacity-data mismatch: larger models can overfit on the limited diversity of small, weakly labeled datasets, whereas smaller models generalize better.
  • Localization vs. Classification: The research confirms that higher classification accuracy doesn't necessarily mean better localization. In fact, MemCAM's localization improved even as its classification accuracy slightly drifted, showing that the consistency loss successfully reshapes spatial activations independently of the class label.

Key Results

MemCAM sets a new state-of-the-art for WS-VSS across both benchmarks:

  • YouTube-Objects v2.2: MemCAM-tiny achieved 76.8% CorLoc, an 8.2% improvement over the single-frame baseline (MCTformer+).
  • $WS^2$ Dataset: MemCAM-tiny reached 41.21% mIoU, outperforming both image-based methods and temporal methods using external cues like optical flow (POF-CAM at 38.70%).

Qualitatively, MemCAM produces masks that are significantly more complete and less "flickery" than baseline methods, as seen in Figure 1.

Figure 1. Qualitative comparison of generated CAMs on the YouTube-Objects v2.2 dataset. (a) Original video frames. (b) Baseline CAMs generated by the single-frame MCTformer+ [38] backbone. (c) CAMs generated by our proposed MemCAM. MemCAM pro- duces superior results, demonstrating more precise localization and reduced background noise, achieving more spatially complete and less fragmented results.

The ablation study (summarized in the table below) highlights that the combination of the Memory Bank (M) and the CAM Consistency Loss ($L_{CAM}$) is crucial; using memory without the specific loss actually degrades performance.

Ablation study showing the impact of Memory and CAM Loss on localization performance.

Limitations and Future Work

  • Teacher Dependency: The quality of the synthetic targets is upper-bounded by the initial quality of the "teacher" (the frozen backbone). If the backbone produces very poor initial CAMs, propagation cannot fully fix them.
  • Abrupt Changes: The feature-space affinities can degrade during very fast motion, heavy occlusions, or shot cuts, where the visual appearance changes too drastically for the memory bank to provide useful context.
  • Memory Horizon: The current FIFO (First-In-First-Out) memory queue is short (typically 5 frames). Objects that leave the scene and return much later are not "remembered."

Future work aims to extend this self-supervisory mechanism to even more challenging regimes, such as fully unsupervised or semi-supervised video semantic segmentation.

Summary updated

Abstract

Weakly Supervised Video Semantic Segmentation (WS-VSS) aims to learn temporally consistent semantic object masks in videos using only video-level labels. While fully supervised models successfully learn temporal relationships by means of cross-attention, training these temporal weights under weak supervision is unstable and even pushes the model below its single-frame baseline. Therefore, existing WS-VSS methods exploit temporal consistency by relying on external priors, such as optical flow. Unfortunately these cues are non-semantic, computationally expensive and noisy, degrading the supervisory signal. Instead of supervising attention weights directly or using external priors, we propose MemCAM, a WS-VSS transformer framework that learns temporal cross-attention directly through feature-space self-supervision. MemCAM introduces a CAM Propagation Algorithm that synthesizes a supervisory signal by propagating past CAMs according to patch-level feature similarities in the latent space. Specifically, we constrain their induced CAMs through a consistency objective between predicted and propagated maps, stabilizing temporal attention learning. Experiments on the YouTube-Objects v2.2 and WS$^2$ datasets demonstrate that MemCAM consistently outperforms the temporal localization and segmentation performance of WS-VSS alternatives, showing that cross-attention can be effectively trained in label-scarce video settings through internal self-supervision.
BibTeX
@misc{marelli2026unlocking,
  author = {Andrea Marelli and Alessandro Ardenghi and Giacomo Boracchi and Luca Magri},
  title = {{Unlocking Cross-Attention in Label-Scarce Regimes: A Weakly Supervised Approach for Video Semantic Segmentation}},
  year = {2026},
  eprint = {2611.18324},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18324},
}
Figure