Abstract
Open-Vocabulary Segmentation (OVS) aims to segment classes that are not present in the training dataset. However, most existing studies assume that the training data is fixed in advance, overlooking more practical scenarios where new datasets are continuously collected over time. To address this, we first analyze how existing OVS models perform under such conditions. In this context, we explore several approaches such as retraining, fine-tuning, and continual learning but find that each of them has clear limitations. To address these issues, we propose ConOVS, a novel continual learning method based on a Mixture-of-Experts framework. ConOVS dynamically combines expert decoders based on the probability that an input sample belongs to the distribution of each incremental dataset. Through extensive experiments, we show that ConOVS consistently outperforms existing methods across pre-training, incremental, and zero-shot test datasets, effectively expanding the recognition capabilities of OVS models when data is collected sequentially.
Introduction: The Challenge of Sequential Data in OVS
Open-Vocabulary Segmentation (OVS) has revolutionized computer vision by allowing models to segment objects beyond their training categories, typically by leveraging large-scale vision-language models like CLIP. However, standard OVS research assumes a static training phase. In real-world applications such as autonomous driving or robotics, data is not static; it arrives in a continuous stream from diverse environments.
When existing OVS models encounter new datasets (e.g., moving from COCO to Cityscapes), they face a significant dilemma. Retraining from scratch is computationally prohibitive and requires access to all historical data. Simple fine-tuning, while easier, leads to catastrophic forgetting. As the model adapts to new data, it loses its ability to segment original classes and, crucially, its "zero-shot" capability to recognize unseen objects. Traditional continual learning (CL) methods often fail here because they are designed for a fixed number of classes, which contradicts the open-vocabulary nature of OVS.
ConOVS: A Mixture-of-Experts Approach
To overcome these limitations, we introduce ConOVS, a framework designed for sustainable OVS. ConOVS utilizes a Mixture-of-Experts (MoE) architecture specifically focused on the model's decoder. Instead of updating the entire model, ConOVS focuses on specialized components:
- Expert Training and Fingerprinting: For each new dataset, ConOVS fine-tunes a dedicated "expert" decoder. To manage these experts, the model generates a "statistical fingerprint" for each dataset by calculating the mean and covariance of image and text embeddings during training.
- Dynamic Interpolation: During inference, an Interpolation Factor Estimator compares the embeddings of a new input sample against the stored fingerprints. It calculates interpolation weights ($\lambda$) that represent the likelihood of the sample belonging to each expert's domain.
- Weight Blending: Instead of simple expert selection, ConOVS blends the weights of the expert decoders relative to the base pre-trained model. This interpolation allows the model to generalize effectively to samples that may fall between known distributions, preserving zero-shot performance while excelling on new data.
Experimental Evaluation
We evaluated ConOVS against several baselines, including standard fine-tuning and established CL methods like EWC and LwF. Our experiments spanned pre-training on COCO, incremental learning on Cityscapes and ADE20K, and zero-shot evaluation on a diverse suite of datasets including LVIS, BDD100K, and Mapillary Vistas.
ConOVS consistently demonstrated superior performance. For instance, when learning Cityscapes after COCO, ConOVS achieved a +20.2 PQ improvement on the new dataset while simultaneously increasing zero-shot performance on ADE20K by +2.5 PQ.
Beyond accuracy, ConOVS is highly efficient. Storing a new expert decoder requires only about 80MB (roughly 6% of the total model size), and the inference overhead is minimal, increasing latency by only ~7% even with multiple experts.
Conclusion and Future Directions
ConOVS provides a scalable and effective solution for continual learning in open-vocabulary segmentation. By leveraging dynamic weight interpolation and statistical fingerprinting, it avoids catastrophic forgetting while expanding the model's knowledge base. Future work includes extending this framework to Open-Vocabulary Object Detection (OVD) and exploring its performance in weakly supervised settings where high-quality labels are scarce.
Summary updated