MISC210K: A Large-Scale Dataset for Multi-Instance Semantic Correspondence

Yixuan Sun, Yiwen Huang, Haijing Guo, Yuzhou Zhao, Runmin Wu, Yizhou Yu, Weifeng Ge, Wenqiang Zhang

Introduction: Beyond Single-Object Matching

Semantic correspondence—the task of identifying matching parts across different objects of the same category—is a fundamental challenge in computer vision. While existing benchmarks like SPair-71k have advanced the field, they predominantly focus on a "single-object schema," where each image contains only one instance of a category. Real-world scenes, however, are rarely so simple. A typical street scene may contain multiple cars, each with its own set of "left headlights" or "front wheels."

Comparison of single-instance matching (left) vs. the multi-instance challenge (right) addressed by MISC210K.

Current models often fail in these "many-to-many" scenarios because they lack the ability to distinguish between individual instances. To bridge this gap, researchers from Fudan University and the University of Hong Kong have introduced MISC210K, the first large-scale dataset specifically designed for multi-instance semantic correspondence.

The MISC210K Dataset

Derived from the COCO Detection 2017 set, MISC210K represents a significant leap in scale and complexity:

  • Scale: It contains 218,179 image pairs, making it roughly three times larger than SPair-71k.
  • Diversity: The dataset covers 34 object classes across 4,812 unique images.
  • Complexity: Images feature up to four instances of the same category, introducing challenges like scale variation, occlusion, and perspective distortion.

Unlike previous datasets that focus on "Single" match objects, MISC210K is built for "Multiple" match objects, requiring models to understand instance boundaries to avoid "confusing" similar parts belonging to different objects.

Methodology: 3D Prototypes and Collaborative Learning

The creation of such a massive, high-quality dataset required a novel approach to annotation and model design.

Robust Annotation Pipeline

To ensure consistency, the authors used 3D models as "prototypes" to define dense keypoints (ranging from 5 to 52 per object). This ensures that every keypoint has a specific, consistent semantic meaning across different 2D views. They implemented a human-machine collaborative workflow where an Automatic Labeling Module (ALM) predicted keypoints, which were then refined by human reviewers.

The MISC210K construction pipeline, including category filtering, 3D prototype definition, and the collaborative annotation interface.

Dual-Path Collaborative Learning (DPCL)

To solve the many-to-many matching problem, the authors developed the Dual-Path Collaborative Learning (DPCL) architecture. This model simultaneously handles two tasks:

  1. Correspondence Path: Generates 4D cost volumes using a transformer-based backbone (iBOT) to represent similarities between all pixel pairs.
  2. Segmentation Path: Predicts instance masks to group matching keypoints. This joint approach ensures the model identifies specific parts of specific instances (e.g., "the front-left wheel of car #2").

Key Insights and Results

The authors established a new benchmark using the mPCK@$\alpha$ (multi-instance Percentage of Correct Keypoints) metric. Key findings include:

  • Superior Performance: The DPCL model outperformed state-of-the-art methods like MMNet and CATs. At a strict threshold ($\alpha=0.05$), DPCL achieved an mPCK of 11.32, nearly doubling the performance of MMNet (5.68).
  • The Necessity of Segmentation: Ablation studies revealed that removing segmentation supervision led to an 8.8% drop in accuracy, proving that understanding instance boundaries is vital for multi-instance matching.
  • Robust Feature Learning: Pre-training on MISC210K improved performance on simpler, single-instance datasets like SPair-71k by up to 2.2%, suggesting that learning in cluttered environments yields more robust feature representations.

Limitations and Future Outlook

Despite these advances, challenges remain. Models still struggle with feature ambiguity between identical-looking instances and the resolution constraints imposed by high-dimensional cost volumes. Future research may focus on "Unseen Keypoint Discovery" and "Closed-loop Constraints" to further enhance self-supervised learning in these complex environments.

Summary updated

Abstract

Semantic correspondence have built up a new way for object recognition. However current single-object matching schema can be hard for discovering commonalities for a category and far from the real-world recognition tasks. To fill this gap, we design the multi-instance semantic correspondence task which aims at constructing the correspondence between multiple objects in an image pair. To support this task, we build a multi-instance semantic correspondence (MISC) dataset from COCO Detection 2017 task called MISC210K. We construct our dataset as three steps: (1) category selection and data cleaning; (2) keypoint design based on 3D models and object description rules; (3) human-machine collaborative annotation. Following these steps, we select 34 classes of objects with 4,812 challenging images annotated via a well designed semi-automatic workflow, and finally acquire 218,179 image pairs with instance masks and instance-level keypoint pairs annotated. We design a dual-path collaborative learning pipeline to train instance-level co-segmentation task and fine-grained level correspondence task together. Benchmark evaluation and further ablation results with detailed analysis are provided with three future directions proposed. Our project is available on https://github.com/YXSUNMADMAX/MISC210K.
BibTeX
@inproceedings{sun2023misc,
  author = {Yixuan Sun and Yiwen Huang and Haijing Guo and Yuzhou Zhao and Runmin Wu and Yizhou Yu and Weifeng Ge and Wenqiang Zhang},
  title = {{MISC210K: A Large-Scale Dataset for Multi-Instance Semantic Correspondence}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00688},
}

Loading related content...

Figure