DMM-Net: Differentiable Mask-Matching Network for Video Object Segmentation

Xiaohui Zeng, Renjie Liao, Li Gu, Yuwen Xiong, Sanja Fidler, Raquel UrtasunICCV 2019 · OpenReview

Abstract

In this paper, we propose the differentiable mask-matching network (DMM-Net) for solving the video object segmentation problem where the initial object masks are provided. Relying on the Mask R-CNN backbone, we extract mask proposals per frame and formulate the matching between object templates and proposals as a linear assignment problem where the cost matrix is predicted by a deep convolutional neural network. We propose a differentiable matching layer which unrolls a projected gradient descent algorithm in which the projection step exploits the Dykstra's algorithm. We prove that under mild conditions, the matching is guaranteed to converge to the optimal one. In practice, it achieves similar performance compared to the Hungarian algorithm during inference. Meanwhile, we can back-propagate through it to learn the cost matrix. After matching, a U-Net style architecture is exploited to refine the matched mask per time step. On DAVIS 2017 dataset, DMM-Net achieves the best performance without online learning on the first frames and the 2nd best with it. Without any fine-tuning, DMM-Net performs comparably to state-of-the-art methods on SegTrack v2 dataset. At last, our differentiable matching layer is very simple to implement; we attach the PyTorch code in the supplementary material which is less than 50 lines long.

Introduction

Video Object Segmentation (VOS) is a fundamental task in computer vision that requires tracking and segmenting specific objects throughout a video sequence, typically guided by a ground-truth mask provided in the initial frame. This task presents significant challenges due to object deformations, rapid motion, occlusions, and the reappearance of objects after they have left the frame.

Existing VOS methods generally fall into two categories: pixel-level matching and mask-level matching. While pixel-level approaches offer high accuracy, they are often computationally expensive and memory-intensive. Conversely, mask-level matching is faster but traditionally relies on non-differentiable algorithms like the Hungarian method or greedy heuristics. These discrete methods prevent end-to-end training of the matching process, forcing models to rely on pre-defined features rather than learned ones optimized for the specific assignment task. DMM-Net addresses this gap by introducing a differentiable matching layer.

Methodology

DMM-Net treats the VOS task as a Linear Assignment Problem (LAP). The pipeline consists of four primary stages: proposal generation, cost matrix calculation, differentiable matching, and mask refinement.

Pipeline Overview

  1. Proposal Generation: The system utilizes a Mask R-CNN backbone to generate a set of potential object masks (proposals) for every frame in the video.
  2. Cost Matrix Calculation: A CNN extracts deep features from both the initial "template" masks and the newly generated proposals. The cost matrix is constructed by computing the cosine similarity between these features, augmented by the Intersection-over-Union (IoU) of the masks to incorporate spatial priors.
  3. Differentiable Matching Layer: This is the core innovation of the paper. Instead of a discrete solver, the authors use Projected Gradient Descent (PGD) to solve the assignment. To satisfy the constraints of a matching matrix (where rows sum to 1 and columns sum to $\le$ 1), they unroll Dykstra’s cyclic projection algorithm. This allows gradients to flow through the matching step, enabling the network to learn the optimal features for matching during the training phase.
  4. Refinement: After a proposal is matched to a template, a ConvLSTM-based refinement head (U-Net style) processes the matched mask to ensure temporal consistency and fine-grained detail.
Overview of the DMM-Net architecture showing the CNN backbone, the matching layer with assignment matrix, and the refinement stage.

Mathematical Convergence

A key contribution of this work is the mathematical proof (Theorem 1) demonstrating that the unrolled matching algorithm is guaranteed to converge to an optimal assignment under mild conditions. This theoretical grounding ensures that the differentiable layer performs as reliably as traditional solvers like the Hungarian algorithm during inference while providing the benefits of end-to-end learnability.

Experimental Results

The authors evaluated DMM-Net on three major benchmarks: YouTube-VOS, DAVIS 2017, and SegTrack v2.

Quantitative Performance

On the DAVIS 2017 dataset, DMM-Net achieved the highest performance among all methods that do not utilize "online learning" (the process of fine-tuning on the first frame at test time). This makes it particularly suitable for real-time applications where the latency of online learning is prohibitive. On YouTube-VOS, the model achieved a competitive 58.0 $\mathcal{G}$-score while operating at approximately double the speed of high-performing competitors like S2S.

Qualitative Analysis

Visual results demonstrate that DMM-Net is robust across a variety of challenging scenarios, including multi-object tracking and complex deformations. The model excels at maintaining the structural integrity of objects even late into long video sequences.

Qualitative results of DMM-Net across various datasets (pandas, dogs, people, fish, cars, and penguins) showing consistent segmentation over time.

Conclusion and Future Work

DMM-Net successfully bridges the gap between discrete assignment algorithms and deep learning. By making the matching layer differentiable, the model learns to extract features that are specifically useful for tracking objects through time.

While highly effective, the model's performance is currently bounded by the quality of the Mask R-CNN proposals; if the backbone fails to detect a small or heavily occluded object, the matching layer cannot recover it. Future research may explore "multi-partite" matching—extending the assignment logic across multiple frames simultaneously—to further enhance temporal stability and handle long-term occlusions more effectively.

Summary updated

Abstract

In this paper, we propose the differentiable mask-matching network (DMM-Net) for solving the video object segmentation problem where the initial object masks are provided. Relying on the Mask R-CNN backbone, we extract mask proposals per frame and formulate the matching between object templates and proposals at one time step as a linear assignment problem where the cost matrix is predicted by a CNN. We propose a differentiable matching layer by unrolling a projected gradient descent algorithm in which the projection exploits the Dykstra's algorithm. We prove that under mild conditions, the matching is guaranteed to converge to the optimum. In practice, it performs similarly to the Hungarian algorithm during inference. Meanwhile, we can back-propagate through it to learn the cost matrix. After matching, a refinement head is leveraged to improve the quality of the matched mask. Our DMM-Net achieves competitive results on the largest video object segmentation dataset YouTube-VOS. On DAVIS 2017, DMM-Net achieves the best performance without online learning on the first frames. Without any fine-tuning, DMM-Net performs comparably to state-of-the-art methods on SegTrack v2 dataset. At last, our matching layer is very simple to implement; we attach the PyTorch code ($<50$ lines) in the supplementary material. Our code is released at https://github.com/ZENGXH/DMM_Net.
BibTeX
@inproceedings{zeng2019dmm,
  author = {Xiaohui Zeng and Renjie Liao and Li Gu and Yuwen Xiong and Sanja Fidler and Raquel Urtasun},
  title = {{DMM-Net: Differentiable Mask-Matching Network for Video Object Segmentation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=As4E1sZVlNP},
}
Figure