Open-Vocabulary Panoptic Segmentation Using Bert Pre-Training of Vision-Language Multiway Transformer Model

Yi-Chia Chen, Wei-Hua Li, Chu-Song Chen

Abstract

Traditional panoptic segmentation models are limited by closed-vocabulary constraints, restricting their utility to a predefined set of object categories. While recent efforts have leveraged CLIP-based architectures to achieve open-vocabulary capabilities, these models often rely on late-stage fusion that neglects deep cross-modal interactions. This paper presents OMTSeg, a novel framework that utilizes the BEiT-3 multiway transformer backbone to enable rich, middle-layer interactions between visual and linguistic features. By incorporating a specialized visual adapter and a multiway segmentation head, OMTSeg achieves state-of-the-art performance on challenging benchmarks like ADE20K, demonstrating superior generalization and efficiency compared to existing large-scale models.

Introduction

Panoptic segmentation aims to provide a comprehensive understanding of a scene by assigning a semantic label and a unique instance ID to every pixel. However, conventional models are typically trained on datasets with a fixed number of classes, such as the 80 categories in COCO. This "closed-vocabulary" nature limits their applicability in real-world scenarios where an infinite variety of objects may be encountered.

Open-vocabulary panoptic segmentation seeks to overcome this by enabling models to recognize and segment any category described in natural language. Current state-of-the-art approaches often employ CLIP (Contrastive Language-Image Pre-training) as a backbone. However, CLIP's dual-encoder architecture processes images and text independently, only aligning them in a shared embedding space at the final layer. This "late fusion" strategy fails to capture the intricate, local-level correspondences between text descriptions and image regions necessary for precise segmentation.

The OMTSeg Framework

OMTSeg (Open-vocabulary Multiway Transformer Segmentation) addresses these limitations by shifting from independent encoders to a unified multiway transformer architecture.

BEiT-3 Multiway Transformer

At the core of OMTSeg is BEiT-3, a foundation model that treats image and text modalities symmetrically. Unlike CLIP, BEiT-3 allows for cross-modal attention within the backbone itself. This enables visual features to be conditioned on linguistic context (and vice versa) throughout the encoding process, facilitating a deeper understanding of how specific words relate to specific visual patterns.

Visual Adapter and Spatial Prior Module

Foundation models like BEiT-3 are generally pre-trained on global image-text pairs, making them less effective for dense, pixel-level tasks. To bridge this gap, OMTSeg introduces a Visual Adapter designed to extract and inject spatial information:

  • Spatial Prior Module (SPM): This module captures local spatial context across multiple resolutions, ensuring the model can identify both large "stuff" (like sky or road) and small "things" (like a distant pedestrian).
  • Spatial Feature Injector (SFI): These priors are injected back into the BEiT-3 layers, allowing the backbone to refine its representations with dense spatial awareness.
  • Multi-Scale Feature Extractor (MSFE): This component further refines the features to handle the inherent scale variance of objects in natural scenes.

Language Prompting and [WLS] Token

To align linguistic descriptions with visual features, OMTSeg utilizes a specialized [WLS] (Word-Level Segment) token to represent category names. Rather than using static, frozen text embeddings, the framework employs prompt tuning. This allows the model to adapt linguistic representations to the specific visual context of the task, improving the mapping between text labels and their corresponding visual masks.

Experimental Results

The model was trained on the COCO Panoptic dataset and evaluated in a zero-shot setting across several diverse benchmarks to test its open-vocabulary capabilities.

Performance on ADE20K

OMTSeg demonstrated significant improvements over existing methods on the ADE20K dataset. On the ADE20K-150 benchmark, it achieved a Panoptic Quality (PQ) of 27.5, substantially outperforming previous leaders such as ODISE (23.4 PQ) and MaskCLIP (15.1 PQ). Furthermore, on the massive ADE20K-847 dataset, which contains nearly 850 categories, OMTSeg reached a mean Intersection over Union (mIoU) of 13.9, setting a new standard for high-granularity open-vocabulary segmentation.

Generalization and Efficiency

The model's robustness was further validated on the Pascal Context and Pascal VOC datasets. Across all variants (e.g., PC-59, PC-459, PAS-20), OMTSeg maintained high performance, indicating that it learns generalized visual-linguistic concepts rather than simply memorizing training data.

Notably, OMTSeg achieves these results with high efficiency. Compared to diffusion-based models like ODISE, which utilize approximately 1.5 billion parameters, OMTSeg operates with 720 million parameters. This reduction in complexity, combined with superior performance on zero-shot benchmarks, highlights the effectiveness of the multiway transformer approach.

Discussion and Conclusion

Ablation studies conducted by the authors underscore the importance of cross-modal interaction. When cross-modal attention was removed from the backbone, the PQ score on ADE20K dropped from 27.5 to 18.4, confirming that early and middle-layer fusion is critical for open-vocabulary tasks.

While OMTSeg represents a significant leap forward, it still faces challenges. For instance, on the original COCO dataset (the training domain), it performs slightly below the much larger ODISE model (54.9 PQ vs. 55.4 PQ). This suggests that while OMTSeg excels at generalization, there is still room to scale the architecture for even higher peak performance.

In conclusion, OMTSeg demonstrates that leveraging a multiway transformer with deep cross-modal interactions is a powerful strategy for open-vocabulary panoptic segmentation. By moving beyond the limitations of late-fusion CLIP models and utilizing efficient visual adapters, OMTSeg provides a scalable and effective framework for understanding complex scenes in an open-world context. Future research may extend this approach to video-based tasks and more complex multimodal reasoning.

Summary updated

Abstract

Open-vocabulary panoptic segmentation remains a challenging problem. One of the biggest difficulties lies in training models to generalize to an unlimited number of classes using limited categorized training data. Recent popular methods involve large-scale vision-language pre-trained foundation models, such as CLIP. In this paper, we propose OMTSeg for open-vocabulary segmentation using another large-scale vision-language pre-trained model called BEiT-3 and leveraging the cross-modal attention between visual and linguistic features in BEiT-3 to achieve better performance. Experiments result demonstrates that OMTSeg performs favorably against state-of-the-art models. Code is available at https://github.com/AI-Application-and-IntegrationLab/OMTSeg
BibTeX
@inproceedings{chen2024open,
  author = {Yi-Chia Chen and Wei-Hua Li and Chu-Song Chen},
  title = {{Open-Vocabulary Panoptic Segmentation Using Bert Pre-Training of Vision-Language Multiway Transformer Model}},
  booktitle = {ICIP},
  year = {2024},
  url = {https://openreview.net/forum?id=3yUbP9H7le},
}
Figure