1st Place Solutions for the UVO Challenge 2022

Jiajun Zhang, Boyu Chen, Zhilong Ji, Jinfeng Bai, Zonghai Hu

Abstract

This paper describes the approach we have taken in the challenge. We still adopted the two-stage scheme same as the last champion, that is, detection first and segmentation followed. We trained more powerful detector and segmentor separately. Besides, we also perform pseudo-label training on the test set, based on student-teacher framework and end-to-end transformer based object detection. The method ranks first on the 2nd Unidentified Video Objects (UVO) challenge, achieving AR@100 of 46.8, 64.7 and 32.2 in the limited data frame track, unlimited data frame track and video track respectively.

Introduction

Standard instance segmentation models typically operate under a "closed-world" assumption, where they are trained to recognize a predefined set of categories. However, real-world applications demand models capable of detecting and segmenting any object, including those never encountered during training. The 2022 Unidentified Video Objects (UVO) Challenge addresses this through the task of Open-World Instance Segmentation. The objective is to develop class-agnostic algorithms that can identify every distinct object in a video frame, regardless of its category.

The challenge is divided into three tracks:

  1. Limited Data Frame Track: Restricted training sets.
  2. Unlimited Data Frame Track: Allows the use of any public data.
  3. Video Track: Requires temporal tracking across frames.

Our solution, which secured first place in all three tracks, utilizes a robust two-stage pipeline combining state-of-the-art architectures with semi-supervised learning techniques.

Methodology

We opted for a decoupled two-stage framework—detection followed by segmentation—finding that it offered greater flexibility and performance than current end-to-end models like ViTDet.

Detection: DINO

For the detection stage, we employed the DINO detector with a Swin-Large (SwinL) backbone. DINO is a Transformer-based detector that enhances the DETR architecture through denoising training and a "look forward twice" scheme for more accurate bounding box prediction.

Segmentation: Mask2Former

The second stage utilizes Mask2Former with a ViT-Adapter-L backbone. Mask2Former treats segmentation as a mask classification problem. The integration of ViT-Adapter allows a standard Vision Transformer to extract the hierarchical features necessary for high-resolution dense prediction tasks.

Semi-Supervised Object Detection (SSOD)

To further enhance performance, particularly on the UVO test set distribution, we implemented a Soft-teacher framework for semi-supervised learning.

Soft-teacher framework for semi-supervised object detection.

As illustrated in Figure 1, this framework involves a "Teacher" model that generates pseudo-labels from weakly augmented unlabeled images. These pseudo-labels, after filtering, are used to supervise a "Student" model trained on strongly augmented versions of the same images. This approach provided a significant +3.9 AR@100 boost in the Limited Data track.

Data Scaling and Training

Generalization to "unidentified" objects requires exposure to diverse data. We utilized several large-scale datasets:

  • UVO Dataset: The core benchmark for open-world video object segmentation.
  • COCO & ImageNet: Standard datasets for initial pre-training.
  • Object365: A massive detection dataset (1.7M images) used to boost performance in the Unlimited track.
  • OpenImage: Used for pre-training the segmentation model on a wide variety of object masks.

We observed that pre-training on Object365 was highly effective; once the model was sufficiently trained on this dataset, the addition of COCO data yielded diminishing returns.

Results

Our approach achieved first place across all tracks of the 2022 UVO Challenge. The final performance metrics (AR@100) were:

  • Limited Data Frame Track: 46.8
  • Unlimited Data Frame Track: 64.7
  • Video Track: 32.2
Performance improvements from architectural upgrades and data scaling on the UVO validation set.

Figure 2 demonstrates the impact of our design choices. In the Unlimited track, upgrading from a 4-scale to a 5-scale DINO model, incorporating Object365 pre-training, and applying pseudo-label finetuning pushed the validation AR@100 from 76.6 to 82.2. Similarly, the foreground IoU improved significantly as more diverse segmentation data (COCO and OpenImage) was introduced.

Discussion and Future Work

Our success highlights the continued efficacy of two-stage frameworks in open-world scenarios, where training specialized, high-capacity models for detection and segmentation separately allows for better generalization.

However, challenges remain in temporal tracking. We found that standard Re-identification (ReID) features in DeepSort often failed to distinguish between different "unknown" objects, leading to identity switches. Future work will focus on improving temporal association for novel categories. Additionally, hardware constraints (32G V100 GPUs) limited our batch sizes during semi-supervised training; larger-scale compute could potentially unlock further performance gains.

Summary updated

Abstract

This paper describes the approach we have taken in the challenge. We still adopted the two-stage scheme same as the last champion, that is, detection first and segmentation followed. We trained more powerful detector and segmentor separately. Besides, we also perform pseudo-label training on the test set, based on student-teacher framework and end-to-end transformer based object detection. The method ranks first on the 2nd Unidentified Video Objects (UVO) challenge, achieving AR@100 of 46.8, 64.7 and 32.2 in the limited data frame track, unlimited data frame track and video track respectively.
BibTeX
@misc{zhang2022st,
  author = {Jiajun Zhang and Boyu Chen and Zhilong Ji and Jinfeng Bai and Zonghai Hu},
  title = {{1st Place Solutions for the UVO Challenge 2022}},
  year = {2022},
  eprint = {2210.09629},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.09629},
}
Figure