OVS Meets Continual Learning: Towards Sustainable Open-Vocabulary Segmentation

Dongjun Hwang, Yejin Kim, Minyoung Lee, Seong Joon Oh, Junsuk Choe

TL;DR

Open-Vocabulary Segmentation (OVS) models excel at segmenting unseen objects, but they assume training data is static. When new datasets are introduced sequentially, standard fine-tuning causes catastrophic forgetting, while retraining from scratch is computationally prohibitive. This paper introduces ConOVS, a novel continual learning framework that dynamically interpolates dataset-specific decoder "experts" based on the statistical distribution of the input, preserving prior knowledge while effectively adapting to new domains.

What Problem Does This Paper Address?

Open-Vocabulary Segmentation (OVS) aims to break the limits of closed-set vision models by segmenting objects belonging to arbitrary, unseen categories. However, current OVS paradigms assume a luxury rarely found in production: a fixed, all-encompassing training dataset. In reality, data arrives sequentially over time (e.g., a robotics system encountering new environments or autonomous driving fleets collecting data in new cities).

If you naively fine-tune an OVS model on new data, it suffers from catastrophic forgetting, losing its ability to segment both the original training categories and completely novel zero-shot classes. Conversely, retraining the model from scratch on all accumulated data is incredibly expensive and often impossible due to data privacy or retention policies. Traditional Continual Learning (CL) methods don't solve this either; they assume a finite, closed-set label space and typically treat unseen classes as "background," which fundamentally breaks the open-vocabulary capability.

Methods

The authors propose ConOVS (Continual Open-Vocabulary Segmentation), a framework based on a Mixture-of-Experts (MoE) approach that updates only the model's decoder while keeping the vision-language encoders (like CLIP) frozen.

Figure 3: Overview of the inference process of our proposed method.

1. Training Phase (Building Experts)

Instead of modifying the base model globally, ConOVS trains a distinct decoder expert for each incoming dataset.

  • The base OVS model is first pre-trained on a base dataset.
  • When a new incremental dataset arrives, only the decoder is fine-tuned, creating a new "expert" decoder.
  • Simultaneously, the model calculates a Multivariate Normal (MVN) distribution (mean and covariance) of the image and text embeddings for that specific dataset to capture its unique statistical signature.

2. Inference Phase (Dynamic Interpolation)

At test time, rather than routing an input image to a single expert (a hard switch), ConOVS performs a soft, dynamic weight interpolation:

  • The input image and text descriptions are passed through the frozen encoders to extract embeddings.
  • The model calculates the likelihood that these embeddings belong to each trained dataset's MVN distribution.
  • A softmax operation normalizes these scores into probabilities, and an element-wise maximum combines the image and text modalities into an interpolation factor ($\lambda$).
  • The final decoder weights are dynamically constructed on the fly by linearly combining the task vectors of the experts based on $\lambda$.

Datasets

The authors evaluated ConOVS across multiple complex sequential learning scenarios using a wide variety of standard segmentation benchmarks:

  • Pre-training Dataset: COCO
  • Incremental Datasets: Cityscapes (urban driving) and ADE20K (diverse indoor/outdoor scenes)
  • Zero-shot Evaluation Datasets: LVIS, BDD100K, Mapillary Vistas, Pascal Context (PC-59, PC-459), Pascal VOC (PAS-20, PAS-21), and ADE20K-847.
  • Robustness/Domain Shift Datasets: GTA5, DarkZurich, and FoggyZurich.

Key Insights and Ideas

  • Fine-Tuning is a Zero-Shot Killer: Fine-tuning an OVS model on a specific domain drastically narrows its focus, severely damaging its generalized zero-shot capabilities on unrelated datasets.
  • Interpolation Over Selection: Traditional MoE methods select one expert per input. ConOVS shows that interpolating the weights of multiple decoders provides a smoother parameter space that generalizes significantly better to completely unseen, out-of-distribution domains.
  • Modality Synergy: Combining both image and text distributions via an element-wise maximum yields a much more accurate estimation of domain proximity than relying on image or text features alone.

Key Results

ConOVS consistently outperformed standard fine-tuning, retraining, and state-of-the-art continual learning methods across pre-training, incremental, and zero-shot test sets.

Figure 1: (a) Comparison of the performance of the OVS model (fc-clip [52]), Retraining, Fine-tuning, and ConOVS against the closed-set segmentation model OneFormer. (b) Performance of the Baseline (fc-clip [52]), Fine-tuning, Retraining, and ConOVS on the pre-training, incremental, and zero-shot test datasets. PQ is used.
  • Mitigating Forgetting: In a scenario where the model was pre-trained on COCO and incrementally trained on Cityscapes, ConOVS maintained COCO performance (+0.3 Panoptic Quality) while boosting Cityscapes performance by +20.2 PQ and improving zero-shot performance on ADE20K by +2.5 PQ.
  • Storage Efficiency: Because ConOVS only saves the decoder weights for each new expert, it only needs to store roughly 6.11% of the total model size (~80MB) per dataset, making it highly scalable compared to full-model ensembles.
  • Truly Unseen Classes: When evaluating explicitly on classes that never appeared in any training data, ConOVS improved performance by a similar margin (+2.3 mIoU on ADE20K), proving that the performance gain is driven by genuine open-vocabulary generalization, not just memorization.

Limitations and Future Work

  • Per-Sample Decoding Bottleneck: Because ConOVS generates a unique set of decoder weights for every single input sample based on its domain proximity, standard batch processing during inference is restricted. While the shared encoder can still process images in batches, the decoding step must be executed individually per sample, which introduces an inference time overhead when batch sizes are large.
  • Pre-training Bias: ConOVS is explicitly designed to preserve prior knowledge. If the initial pre-training dataset contains severe biases, the model will faithfully preserve those biases throughout its lifecycle.

Summary updated

Abstract

Open-Vocabulary Segmentation (OVS) aims to segment classes that are not present in the training dataset. However, most existing studies assume that the training data is fixed in advance, overlooking more practical scenarios where new datasets are continuously collected over time. To address this, we first analyze how existing OVS models perform under such conditions. In this context, we explore several approaches such as retraining, fine-tuning, and continual learning but find that each of them has clear limitations. To address these issues, we propose ConOVS, a novel continual learning method based on a Mixture-of-Experts framework. ConOVS dynamically combines expert decoders based on the probability that an input sample belongs to the distribution of each incremental dataset. Through extensive experiments, we show that ConOVS consistently outperforms existing methods across pre-training, incremental, and zero-shot test datasets, effectively expanding the recognition capabilities of OVS models when data is collected sequentially.
BibTeX
@inproceedings{hwang2025ovs,
  author = {Dongjun Hwang and Yejin Kim and Minyoung Lee and Seong Joon Oh and Junsuk Choe},
  title = {{OVS Meets Continual Learning: Towards Sustainable Open-Vocabulary Segmentation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=y8Hv7EdcRF},
}
Figure