Abstract
In recent years, open-vocabulary (OV) dense visual prediction (such as OV object detection, semantic, instance and panoptic segmentations) has attracted increasing research attention. However, most of existing approaches are task-specific and individually tackle each task. In this paper, we propose a Unified Open-Vocabulary Network (UOVN) to jointly address four common dense prediction tasks. Compared with separate models, a unified network is more desirable for diverse industrial applications. Moreover, OV dense prediction training data is relatively less. Separate networks can only leverage task-relevant training data, while a unified approach can integrate diverse training data to boost individual tasks. We address two major challenges in unified OV prediction. Firstly, unlike unified methods for fixed-set predictions, OV networks are usually trained with multi-modal data. Therefore, we propose a multi-modal, multi-scale and multi-task (MMM) decoding mechanism to better leverage multi-modal data. Secondly, because UOVN uses data from different tasks for training, there are significant domain and task gaps. We present a UOVN training mechanism to reduce such gaps. Experiments on four datasets demonstrate the effectiveness of our UOVN.
Introduction
Dense visual prediction tasks, including object detection and various forms of segmentation (semantic, instance, and panoptic), are fundamental to computer vision. Traditionally, these tasks are handled by "closed-set" models restricted to a predefined set of categories. Open-Vocabulary (OV) models aim to overcome this by enabling the recognition of arbitrary categories through natural language queries.
However, current OV research faces two significant challenges: task fragmentation and data scarcity. Most existing models are specialized for a single task, which is inefficient for complex applications like autonomous systems. Furthermore, high-quality pixel-level annotations for OV tasks are scarce. While diverse datasets exist (e.g., detection-only or grounding datasets), separate models cannot effectively pool these resources to improve performance across the board.
Methods
The Unified Open-Vocabulary Network (UOVN) addresses these issues by framing all four dense prediction tasks as a unified mask classification problem.
MMM Decoding Mechanism
The core of UOVN is the Multi-modal, Multi-scale, and Multi-task (MMM) decoding mechanism. This component generates bounding boxes and segmentation masks simultaneously. A key innovation within this decoder is the Multi-modal Multi-scale Deformable Attention (MMDA) module. Unlike standard attention mechanisms that rely solely on visual cues, MMDA incorporates language features to guide the visual search. By predicting language-specific offsets, the model focuses its attention on regions most relevant to the text query.
UOVN Training and Adaptation
To leverage heterogeneous datasets, UOVN employs a specialized training mechanism. Since different datasets may not share identical category labels, the authors introduced Global and Local Adaptation Losses. These losses use the semantic similarity between language queries to align visual features across different domains. This allows the model to learn from a mix of grounding, detection, and segmentation data, effectively bridging domain and task gaps.
Results and Insights
UOVN demonstrates that joint training provides a synergistic effect across tasks. For instance, incorporating a detection branch into a segmentation model improved instance segmentation performance by 5.4% and panoptic results by 3.4%.
Performance Benchmarks
Using a standard ResNet-50 backbone, UOVN achieved state-of-the-art or highly competitive results:
- COCO Cross-Dataset: Achieved 41.3 mAP in detection and 30.5 mAP in instance segmentation, outperforming previous unified models like GLIP.
- LVIS Zero-Shot: On rare, unseen classes, it reached 22.6 mAP for detection and 22.0 mAP for instance segmentation, demonstrating strong generalization.
- Segmentation Excellence: On the ADE20k dataset, UOVN outperformed specialized OV segmentation models such as OpenSeg and MaskCLIP.
Conclusion
UOVN represents a significant step toward universal vision models by unifying four major dense prediction tasks into a single framework. By leveraging multi-modal data and cross-task training, it overcomes the limitations of task-specific models and data scarcity. Future work will explore scaling the backbone architecture and training on even larger "foundation-scale" datasets to further enhance open-vocabulary performance.
Summary updated