Abstract
Modern Video Object Segmentation (VOS) frameworks rely heavily on memory-matching mechanisms to track objects across frames. However, these systems often suffer from "false matching," where similar-looking objects or rapid movements lead to accumulated errors and object confusion. This paper introduces JIMD (Jointly Improve Matching and Decoding), a framework that shifts the paradigm from focusing solely on matching accuracy to a joint optimization of both matching and decoding. By implementing a "look and think twice" strategy, JIMD recovers lost spatial details and suppresses matching errors, achieving state-of-the-art performance on the DAVIS and YouTubeVOS benchmarks.
Introduction
Semi-supervised Video Object Segmentation (VOS) involves tracking and masking specific objects throughout a video sequence, given only the initial mask in the first frame. The current state-of-the-art approach typically utilizes a "memory bank" to store features from previous frames. When a new query frame arrives, the model performs a matching operation between the query pixels and the memory bank to determine object membership.
Despite the success of memory-based methods, a critical bottleneck remains: memory matching is inherently prone to error. Even high-performance matching algorithms produce "false matches"—subtle inaccuracies that accumulate over time, resulting in object swapping or fragmented masks. Traditionally, research has focused on refining the matching algorithm itself, treating the decoder—the component responsible for generating the final pixel-level mask—as a passive recipient of matched features. This paper argues that the decoder should instead play an active role in error correction and detail recovery.
The JIMD Framework
The proposed JIMD framework addresses the limitations of traditional VOS by introducing three novel components designed to improve both the temporal matching of features and the spatial refinement of the final output.
1. Cost-Aware Matching (Short-term)
Standard short-term matching often relies on local correlation windows to maintain efficiency. However, this can fail if an object moves rapidly out of the window's range. JIMD introduces a global 4D cost volume between the current frame and the immediate previous frame. By using learnable tokens to extract fine-grained motion details from this volume, the model becomes significantly more robust to rapid motion and sudden changes in object appearance.
2. Cross-Scale Matching (Long-term)
Objects in video sequences frequently undergo scale changes as they move toward or away from the camera. To handle this, JIMD "shunts" long-term memory keys and values into three distinct spatial scales. This multi-scale approach ensures that the model can effectively match objects regardless of whether they appear as large, foreground elements or small, distant figures.
3. Compensatory Decoding
The core of the "look and think twice" philosophy is the Compensatory Decoding module. Rather than simply upsampling matched features, the decoder performs a three-step refinement process:
- Pre-decoding: Generates an initial spatial prompt based on matched features.
- Context Embedding: The model re-examines the original query frame. This allows it to recover low-level details, such as sharp edges and textures, which are often lost or blurred during the abstract matching phase.
- Post-decoding: The model fuses the original image features with the matched features to produce a refined, high-fidelity mask.
This mechanism allows the decoder to act as a safety net, suppressing false matches that do not align with the visual evidence present in the current frame.
Experimental Results
JIMD was evaluated on the gold-standard benchmarks for VOS: DAVIS (2016/2017) and YouTubeVOS (2018/2019).
Quantitative Performance
JIMD achieved top-tier results across all major metrics:
- DAVIS 2017 Test-dev: The model achieved 83.9% J&F, a significant improvement over the widely used XMem (81.0%).
- DAVIS 2017 Val: Reached a high-water mark of 88.1% J&F.
- YouTubeVOS: The model demonstrated strong generalization to unseen object categories, achieving G-scores of 84.8% (2018) and 84.6% (2019).
Qualitative Improvements
Visual analysis confirms that JIMD excels in scenarios where previous methods fail. Specifically, it maintains distinct masks for similar-looking objects in close proximity—such as multiple people wearing similar clothing—where earlier state-of-the-art methods often merged the objects or swapped their identities.
Discussion and Limitations
A key insight of this work is the "plug-and-play" potential of the Compensatory Decoding module. The authors demonstrated that adding this module to existing architectures like STM or TransVOS consistently improved their performance by 1.2% to 1.4% J&F, proving that the bottleneck indeed lies in the interaction between matching and decoding.
However, these improvements come with a computational trade-off. The "readout twice" mechanism and the generation of global cost volumes impact inference speed. JIMD operates at 13.2 FPS on high-end hardware. While this is efficient enough for many applications, it is slower than "readout once" models like XMem, which can reach 22.6 FPS, making JIMD less suitable for real-time constraints on mobile or edge devices.
Conclusion
JIMD demonstrates that memory matching is not a standalone solution for Video Object Segmentation. By jointly optimizing the matching and decoding stages, the framework effectively mitigates the problem of false matching and recovers fine-grained spatial details. This joint paradigm sets a new standard for VOS performance and offers a promising direction for other video-based tasks, such as multi-object tracking and video instance segmentation.
Summary updated