Continuous Adaptation for Interactive Segmentation Using Teacher-Student Architecture

Barsegh Atanyan, Levon Khachatryan, Shant Navasardyan, Yunchao Wei, Humphrey ShiWACV 2024 · OpenReview

Abstract

Interactive segmentation is a critical task where models extract objects of interest based on iterative user feedback, such as clicks. While current state-of-the-art methods perform well on in-distribution data, they often fail to generalize to unseen domains like medical or aerial imagery. Test-time adaptation can address this, but it typically leads to "catastrophic forgetting," where the model loses its original capabilities. We propose a novel teacher-student learning framework for continuous adaptation. By updating a student model via direct interaction and a teacher model via Exponential Moving Average (EMA), our approach enables seamless learning across sequential domains. Our evaluation demonstrates significant improvements in segmentation accuracy on new domains while reducing catastrophic forgetting on the source domain from 55% to 4%.

Introduction

Interactive segmentation allows users to provide "clicks" (positive for foreground, negative for background) to guide a model in generating accurate masks. While modern models trained on standard datasets like COCO are powerful, they struggle with specialized domains—such as segmenting organs in CT scans or buildings in satellite imagery.

To bridge this gap, researchers use test-time adaptation to update the model as the user provides feedback. However, this often results in catastrophic forgetting: as the model learns the nuances of medical images, it loses its ability to segment natural objects. This degradation is a major bottleneck for real-world pipelines where users may label diverse datasets sequentially. This paper introduces a robust architecture to facilitate continuous learning without sacrificing prior knowledge.

Methodology: The Teacher-Student Framework

The proposed architecture balances plasticity (learning new features) and stability (retaining old knowledge) through a dual-model system.

1. Dual-Model Architecture

Both the Student and Teacher models are initialized with the same pre-trained weights. The Student acts as the active learner, while the Teacher serves as a stable knowledge anchor.

2. Student Updates

The Student model undergoes two distinct update phases during interaction:

  • Per-Click Update: After each user click, the Student adjusts its weights to minimize a sparse cross-entropy loss ($L_C$), ensuring immediate responsiveness to user input.
  • Per-Image Update: Once the user completes an image, a final refinement step ($L_I$) is performed using the final prediction and cumulative clicks as pseudo-ground truth.

3. Teacher Stability via EMA

The Teacher is never updated via direct gradient descent. Instead, its weights are updated using an Exponential Moving Average (EMA) of the Student’s weights. This slow, smoothed update allows the Teacher to integrate new domain knowledge without being swayed by the noise of a single image.

4. Continuous Adaptation Cycle

When encountering a new dataset, the Student is re-initialized with the Teacher’s current weights. This ensures the Student always starts a new task with the most stable and comprehensive version of the model's accumulated knowledge.

Experimental Results

The framework was evaluated across a sequence of diverse domains, including medical eye fundus images (DRIONS-DB), aerial imagery (Rooftop), and 3D CT scans (Medical Segmentation Decathlon).

Qualitative comparison of interactive segmentation results across different domains. The proposed method ("Ours") achieves masks closer to the Ground Truth with fewer or more effective user clicks compared to the baseline and the original FocalClick model.

Performance on Unseen Domains

The primary metric used is NoC@90 (the average Number of Clicks required to reach a 90% Intersection over Union score). On the DRIONS-DB medical dataset, the proposed method achieved a NoC@90 of 2.22, a significant improvement over the 6.23 required by the frozen FocalClick baseline.

Mitigating Catastrophic Forgetting

In a sequence where the model adapted to medical and aerial data and was then tested back on the Berkeley dataset, standard adaptation techniques saw performance decline by 55%. Our teacher-student framework reduced this performance decline to just 4%, proving its effectiveness in preserving source-domain knowledge.

Discussion and Limitations

While the framework is backbone-agnostic—performing well with both SegFormer and HRNet—it does introduce computational overhead. Performing gradient descent updates during an interactive session can add latency, which may be a concern for high-resolution production environments. Additionally, the success of the EMA depends on hyperparameter tuning (smoothing factor $\alpha$ and regularization weight $\gamma$), which may require adjustment for vastly different domains.

Conclusion

We have presented a teacher-student architecture that enables continuous, click-based adaptation for interactive segmentation. By leveraging EMA updates and a dual-stage student training process, the model achieves high accuracy on unseen domains while effectively neutralizing catastrophic forgetting. This framework paves the way for more flexible and intelligent annotation tools that evolve alongside the user's needs.

Summary updated

Abstract

Interactive segmentation is the task of segmenting objects or regions of interest from images based on user annotations. While most current methods perform effectively on images from the same distribution as the training dataset, they suffer to generalize on unseen domains. To address this issue some approaches incorporate test-time adaptation techniques which, on the other hand, may lead to catastrophic forgetting (i.e. degrading the performance on the previously seen domains) when applied on datasets from various domains sequentially. In this paper, we propose a novel domain adaptation approach leveraging a teacher-student learning framework to tackle the catastrophic forgetting issue. Continuously updating the student and teacher models based on user clicks results in improved segmentation accuracy on unseen domains, while preserving comparable performance on previous domains. Our approach is evaluated on a sequence of datasets from unseen domains (i.e. medical, aerial images, etc.), and, after adaptation, on the source domain demonstrating a significant decline of catastrophic forgetting (e.g. from 55% to 4% on Berkeley dataset).
BibTeX
@inproceedings{atanyan2024continuous,
  author = {Barsegh Atanyan and Levon Khachatryan and Shant Navasardyan and Yunchao Wei and Humphrey Shi},
  title = {{Continuous Adaptation for Interactive Segmentation Using Teacher-Student Architecture}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=yE81zZTLsg},
}
Figure