Abstract
Traditional panoptic segmentation models are limited by closed-vocabulary constraints, restricting their utility to a predefined set of object categories. While recent efforts have leveraged CLIP-based architectures to achieve open-vocabulary capabilities, these models often rely on late-stage fusion that neglects deep cross-modal interactions. This paper presents OMTSeg, a novel framework that utilizes the BEiT-3 multiway transformer backbone to enable rich, middle-layer interactions between visual and linguistic features. By incorporating a specialized visual adapter and a multiway segmentation head, OMTSeg achieves state-of-the-art performance on challenging benchmarks like ADE20K, demonstrating superior generalization and efficiency compared to existing large-scale models.
Introduction
Panoptic segmentation aims to provide a comprehensive understanding of a scene by assigning a semantic label and a unique instance ID to every pixel. However, conventional models are typically trained on datasets with a fixed number of classes, such as the 80 categories in COCO. This "closed-vocabulary" nature limits their applicability in real-world scenarios where an infinite variety of objects may be encountered.
Open-vocabulary panoptic segmentation seeks to overcome this by enabling models to recognize and segment any category described in natural language. Current state-of-the-art approaches often employ CLIP (Contrastive Language-Image Pre-training) as a backbone. However, CLIP's dual-encoder architecture processes images and text independently, only aligning them in a shared embedding space at the final layer. This "late fusion" strategy fails to capture the intricate, local-level correspondences between text descriptions and image regions necessary for precise segmentation.
The OMTSeg Framework
OMTSeg (Open-vocabulary Multiway Transformer Segmentation) addresses these limitations by shifting from independent encoders to a unified multiway transformer architecture.
BEiT-3 Multiway Transformer
At the core of OMTSeg is BEiT-3, a foundation model that treats image and text modalities symmetrically. Unlike CLIP, BEiT-3 allows for cross-modal attention within the backbone itself. This enables visual features to be conditioned on linguistic context (and vice versa) throughout the encoding process, facilitating a deeper understanding of how specific words relate to specific visual patterns.
Visual Adapter and Spatial Prior Module
Foundation models like BEiT-3 are generally pre-trained on global image-text pairs, making them less effective for dense, pixel-level tasks. To bridge this gap, OMTSeg introduces a Visual Adapter designed to extract and inject spatial information:
- Spatial Prior Module (SPM): This module captures local spatial context across multiple resolutions, ensuring the model can identify both large "stuff" (like sky or road) and small "things" (like a distant pedestrian).
- Spatial Feature Injector (SFI): These priors are injected back into the BEiT-3 layers, allowing the backbone to refine its representations with dense spatial awareness.
- Multi-Scale Feature Extractor (MSFE): This component further refines the features to handle the inherent scale variance of objects in natural scenes.
Language Prompting and [WLS] Token
To align linguistic descriptions with visual features, OMTSeg utilizes a specialized [WLS] (Word-Level Segment) token to represent category names. Rather than using static, frozen text embeddings, the framework employs prompt tuning. This allows the model to adapt linguistic representations to the specific visual context of the task, improving the mapping between text labels and their corresponding visual masks.
Experimental Results
The model was trained on the COCO Panoptic dataset and evaluated in a zero-shot setting across several diverse benchmarks to test its open-vocabulary capabilities.
Performance on ADE20K
OMTSeg demonstrated significant improvements over existing methods on the ADE20K dataset. On the ADE20K-150 benchmark, it achieved a Panoptic Quality (PQ) of 27.5, substantially outperforming previous leaders such as ODISE (23.4 PQ) and MaskCLIP (15.1 PQ). Furthermore, on the massive ADE20K-847 dataset, which contains nearly 850 categories, OMTSeg reached a mean Intersection over Union (mIoU) of 13.9, setting a new standard for high-granularity open-vocabulary segmentation.
Generalization and Efficiency
The model's robustness was further validated on the Pascal Context and Pascal VOC datasets. Across all variants (e.g., PC-59, PC-459, PAS-20), OMTSeg maintained high performance, indicating that it learns generalized visual-linguistic concepts rather than simply memorizing training data.
Notably, OMTSeg achieves these results with high efficiency. Compared to diffusion-based models like ODISE, which utilize approximately 1.5 billion parameters, OMTSeg operates with 720 million parameters. This reduction in complexity, combined with superior performance on zero-shot benchmarks, highlights the effectiveness of the multiway transformer approach.
Discussion and Conclusion
Ablation studies conducted by the authors underscore the importance of cross-modal interaction. When cross-modal attention was removed from the backbone, the PQ score on ADE20K dropped from 27.5 to 18.4, confirming that early and middle-layer fusion is critical for open-vocabulary tasks.
While OMTSeg represents a significant leap forward, it still faces challenges. For instance, on the original COCO dataset (the training domain), it performs slightly below the much larger ODISE model (54.9 PQ vs. 55.4 PQ). This suggests that while OMTSeg excels at generalization, there is still room to scale the architecture for even higher peak performance.
In conclusion, OMTSeg demonstrates that leveraging a multiway transformer with deep cross-modal interactions is a powerful strategy for open-vocabulary panoptic segmentation. By moving beyond the limitations of late-fusion CLIP models and utilizing efficient visual adapters, OMTSeg provides a scalable and effective framework for understanding complex scenes in an open-world context. Future research may extend this approach to video-based tasks and more complex multimodal reasoning.
Summary updated