Abstract
In this paper, we propose the differentiable mask-matching network (DMM-Net) for solving the video object segmentation problem where the initial object masks are provided. Relying on the Mask R-CNN backbone, we extract mask proposals per frame and formulate the matching between object templates and proposals as a linear assignment problem where the cost matrix is predicted by a deep convolutional neural network. We propose a differentiable matching layer which unrolls a projected gradient descent algorithm in which the projection step exploits the Dykstra's algorithm. We prove that under mild conditions, the matching is guaranteed to converge to the optimal one. In practice, it achieves similar performance compared to the Hungarian algorithm during inference. Meanwhile, we can back-propagate through it to learn the cost matrix. After matching, a U-Net style architecture is exploited to refine the matched mask per time step. On DAVIS 2017 dataset, DMM-Net achieves the best performance without online learning on the first frames and the 2nd best with it. Without any fine-tuning, DMM-Net performs comparably to state-of-the-art methods on SegTrack v2 dataset. At last, our differentiable matching layer is very simple to implement; we attach the PyTorch code in the supplementary material which is less than 50 lines long.
Introduction
Video Object Segmentation (VOS) is a fundamental task in computer vision that requires tracking and segmenting specific objects throughout a video sequence, typically guided by a ground-truth mask provided in the initial frame. This task presents significant challenges due to object deformations, rapid motion, occlusions, and the reappearance of objects after they have left the frame.
Existing VOS methods generally fall into two categories: pixel-level matching and mask-level matching. While pixel-level approaches offer high accuracy, they are often computationally expensive and memory-intensive. Conversely, mask-level matching is faster but traditionally relies on non-differentiable algorithms like the Hungarian method or greedy heuristics. These discrete methods prevent end-to-end training of the matching process, forcing models to rely on pre-defined features rather than learned ones optimized for the specific assignment task. DMM-Net addresses this gap by introducing a differentiable matching layer.
Methodology
DMM-Net treats the VOS task as a Linear Assignment Problem (LAP). The pipeline consists of four primary stages: proposal generation, cost matrix calculation, differentiable matching, and mask refinement.
Pipeline Overview
- Proposal Generation: The system utilizes a Mask R-CNN backbone to generate a set of potential object masks (proposals) for every frame in the video.
- Cost Matrix Calculation: A CNN extracts deep features from both the initial "template" masks and the newly generated proposals. The cost matrix is constructed by computing the cosine similarity between these features, augmented by the Intersection-over-Union (IoU) of the masks to incorporate spatial priors.
- Differentiable Matching Layer: This is the core innovation of the paper. Instead of a discrete solver, the authors use Projected Gradient Descent (PGD) to solve the assignment. To satisfy the constraints of a matching matrix (where rows sum to 1 and columns sum to $\le$ 1), they unroll Dykstra’s cyclic projection algorithm. This allows gradients to flow through the matching step, enabling the network to learn the optimal features for matching during the training phase.
- Refinement: After a proposal is matched to a template, a ConvLSTM-based refinement head (U-Net style) processes the matched mask to ensure temporal consistency and fine-grained detail.
Mathematical Convergence
A key contribution of this work is the mathematical proof (Theorem 1) demonstrating that the unrolled matching algorithm is guaranteed to converge to an optimal assignment under mild conditions. This theoretical grounding ensures that the differentiable layer performs as reliably as traditional solvers like the Hungarian algorithm during inference while providing the benefits of end-to-end learnability.
Experimental Results
The authors evaluated DMM-Net on three major benchmarks: YouTube-VOS, DAVIS 2017, and SegTrack v2.
Quantitative Performance
On the DAVIS 2017 dataset, DMM-Net achieved the highest performance among all methods that do not utilize "online learning" (the process of fine-tuning on the first frame at test time). This makes it particularly suitable for real-time applications where the latency of online learning is prohibitive. On YouTube-VOS, the model achieved a competitive 58.0 $\mathcal{G}$-score while operating at approximately double the speed of high-performing competitors like S2S.
Qualitative Analysis
Visual results demonstrate that DMM-Net is robust across a variety of challenging scenarios, including multi-object tracking and complex deformations. The model excels at maintaining the structural integrity of objects even late into long video sequences.
Conclusion and Future Work
DMM-Net successfully bridges the gap between discrete assignment algorithms and deep learning. By making the matching layer differentiable, the model learns to extract features that are specifically useful for tracking objects through time.
While highly effective, the model's performance is currently bounded by the quality of the Mask R-CNN proposals; if the backbone fails to detect a small or heavily occluded object, the matching layer cannot recover it. Future research may explore "multi-partite" matching—extending the assignment logic across multiple frames simultaneously—to further enhance temporal stability and handle long-term occlusions more effectively.
Summary updated