TL;DR
Open-Vocabulary Segmentation (OVS) models excel at segmenting unseen objects, but they assume training data is static. When new datasets are introduced sequentially, standard fine-tuning causes catastrophic forgetting, while retraining from scratch is computationally prohibitive. This paper introduces ConOVS, a novel continual learning framework that dynamically interpolates dataset-specific decoder "experts" based on the statistical distribution of the input, preserving prior knowledge while effectively adapting to new domains.
What Problem Does This Paper Address?
Open-Vocabulary Segmentation (OVS) aims to break the limits of closed-set vision models by segmenting objects belonging to arbitrary, unseen categories. However, current OVS paradigms assume a luxury rarely found in production: a fixed, all-encompassing training dataset. In reality, data arrives sequentially over time (e.g., a robotics system encountering new environments or autonomous driving fleets collecting data in new cities).
If you naively fine-tune an OVS model on new data, it suffers from catastrophic forgetting, losing its ability to segment both the original training categories and completely novel zero-shot classes. Conversely, retraining the model from scratch on all accumulated data is incredibly expensive and often impossible due to data privacy or retention policies. Traditional Continual Learning (CL) methods don't solve this either; they assume a finite, closed-set label space and typically treat unseen classes as "background," which fundamentally breaks the open-vocabulary capability.
Methods
The authors propose ConOVS (Continual Open-Vocabulary Segmentation), a framework based on a Mixture-of-Experts (MoE) approach that updates only the model's decoder while keeping the vision-language encoders (like CLIP) frozen.
1. Training Phase (Building Experts)
Instead of modifying the base model globally, ConOVS trains a distinct decoder expert for each incoming dataset.
- The base OVS model is first pre-trained on a base dataset.
- When a new incremental dataset arrives, only the decoder is fine-tuned, creating a new "expert" decoder.
- Simultaneously, the model calculates a Multivariate Normal (MVN) distribution (mean and covariance) of the image and text embeddings for that specific dataset to capture its unique statistical signature.
2. Inference Phase (Dynamic Interpolation)
At test time, rather than routing an input image to a single expert (a hard switch), ConOVS performs a soft, dynamic weight interpolation:
- The input image and text descriptions are passed through the frozen encoders to extract embeddings.
- The model calculates the likelihood that these embeddings belong to each trained dataset's MVN distribution.
- A softmax operation normalizes these scores into probabilities, and an element-wise maximum combines the image and text modalities into an interpolation factor ($\lambda$).
- The final decoder weights are dynamically constructed on the fly by linearly combining the task vectors of the experts based on $\lambda$.
Datasets
The authors evaluated ConOVS across multiple complex sequential learning scenarios using a wide variety of standard segmentation benchmarks:
- Pre-training Dataset: COCO
- Incremental Datasets: Cityscapes (urban driving) and ADE20K (diverse indoor/outdoor scenes)
- Zero-shot Evaluation Datasets: LVIS, BDD100K, Mapillary Vistas, Pascal Context (PC-59, PC-459), Pascal VOC (PAS-20, PAS-21), and ADE20K-847.
- Robustness/Domain Shift Datasets: GTA5, DarkZurich, and FoggyZurich.
Key Insights and Ideas
- Fine-Tuning is a Zero-Shot Killer: Fine-tuning an OVS model on a specific domain drastically narrows its focus, severely damaging its generalized zero-shot capabilities on unrelated datasets.
- Interpolation Over Selection: Traditional MoE methods select one expert per input. ConOVS shows that interpolating the weights of multiple decoders provides a smoother parameter space that generalizes significantly better to completely unseen, out-of-distribution domains.
- Modality Synergy: Combining both image and text distributions via an element-wise maximum yields a much more accurate estimation of domain proximity than relying on image or text features alone.
Key Results
ConOVS consistently outperformed standard fine-tuning, retraining, and state-of-the-art continual learning methods across pre-training, incremental, and zero-shot test sets.
- Mitigating Forgetting: In a scenario where the model was pre-trained on COCO and incrementally trained on Cityscapes, ConOVS maintained COCO performance (+0.3 Panoptic Quality) while boosting Cityscapes performance by +20.2 PQ and improving zero-shot performance on ADE20K by +2.5 PQ.
- Storage Efficiency: Because ConOVS only saves the decoder weights for each new expert, it only needs to store roughly 6.11% of the total model size (~80MB) per dataset, making it highly scalable compared to full-model ensembles.
- Truly Unseen Classes: When evaluating explicitly on classes that never appeared in any training data, ConOVS improved performance by a similar margin (+2.3 mIoU on ADE20K), proving that the performance gain is driven by genuine open-vocabulary generalization, not just memorization.
Limitations and Future Work
- Per-Sample Decoding Bottleneck: Because ConOVS generates a unique set of decoder weights for every single input sample based on its domain proximity, standard batch processing during inference is restricted. While the shared encoder can still process images in batches, the decoding step must be executed individually per sample, which introduces an inference time overhead when batch sizes are large.
- Pre-training Bias: ConOVS is explicitly designed to preserve prior knowledge. If the initial pre-training dataset contains severe biases, the model will faithfully preserve those biases throughout its lifecycle.
Summary updated