Abstract
Interactive segmentation is a critical task where models extract objects of interest based on iterative user feedback, such as clicks. While current state-of-the-art methods perform well on in-distribution data, they often fail to generalize to unseen domains like medical or aerial imagery. Test-time adaptation can address this, but it typically leads to "catastrophic forgetting," where the model loses its original capabilities. We propose a novel teacher-student learning framework for continuous adaptation. By updating a student model via direct interaction and a teacher model via Exponential Moving Average (EMA), our approach enables seamless learning across sequential domains. Our evaluation demonstrates significant improvements in segmentation accuracy on new domains while reducing catastrophic forgetting on the source domain from 55% to 4%.
Introduction
Interactive segmentation allows users to provide "clicks" (positive for foreground, negative for background) to guide a model in generating accurate masks. While modern models trained on standard datasets like COCO are powerful, they struggle with specialized domains—such as segmenting organs in CT scans or buildings in satellite imagery.
To bridge this gap, researchers use test-time adaptation to update the model as the user provides feedback. However, this often results in catastrophic forgetting: as the model learns the nuances of medical images, it loses its ability to segment natural objects. This degradation is a major bottleneck for real-world pipelines where users may label diverse datasets sequentially. This paper introduces a robust architecture to facilitate continuous learning without sacrificing prior knowledge.
Methodology: The Teacher-Student Framework
The proposed architecture balances plasticity (learning new features) and stability (retaining old knowledge) through a dual-model system.
1. Dual-Model Architecture
Both the Student and Teacher models are initialized with the same pre-trained weights. The Student acts as the active learner, while the Teacher serves as a stable knowledge anchor.
2. Student Updates
The Student model undergoes two distinct update phases during interaction:
- Per-Click Update: After each user click, the Student adjusts its weights to minimize a sparse cross-entropy loss ($L_C$), ensuring immediate responsiveness to user input.
- Per-Image Update: Once the user completes an image, a final refinement step ($L_I$) is performed using the final prediction and cumulative clicks as pseudo-ground truth.
3. Teacher Stability via EMA
The Teacher is never updated via direct gradient descent. Instead, its weights are updated using an Exponential Moving Average (EMA) of the Student’s weights. This slow, smoothed update allows the Teacher to integrate new domain knowledge without being swayed by the noise of a single image.
4. Continuous Adaptation Cycle
When encountering a new dataset, the Student is re-initialized with the Teacher’s current weights. This ensures the Student always starts a new task with the most stable and comprehensive version of the model's accumulated knowledge.
Experimental Results
The framework was evaluated across a sequence of diverse domains, including medical eye fundus images (DRIONS-DB), aerial imagery (Rooftop), and 3D CT scans (Medical Segmentation Decathlon).
Performance on Unseen Domains
The primary metric used is NoC@90 (the average Number of Clicks required to reach a 90% Intersection over Union score). On the DRIONS-DB medical dataset, the proposed method achieved a NoC@90 of 2.22, a significant improvement over the 6.23 required by the frozen FocalClick baseline.
Mitigating Catastrophic Forgetting
In a sequence where the model adapted to medical and aerial data and was then tested back on the Berkeley dataset, standard adaptation techniques saw performance decline by 55%. Our teacher-student framework reduced this performance decline to just 4%, proving its effectiveness in preserving source-domain knowledge.
Discussion and Limitations
While the framework is backbone-agnostic—performing well with both SegFormer and HRNet—it does introduce computational overhead. Performing gradient descent updates during an interactive session can add latency, which may be a concern for high-resolution production environments. Additionally, the success of the EMA depends on hyperparameter tuning (smoothing factor $\alpha$ and regularization weight $\gamma$), which may require adjustment for vastly different domains.
Conclusion
We have presented a teacher-student architecture that enables continuous, click-based adaptation for interactive segmentation. By leveraging EMA updates and a dual-stage student training process, the model achieves high accuracy on unseen domains while effectively neutralizing catastrophic forgetting. This framework paves the way for more flexible and intelligent annotation tools that evolve alongside the user's needs.
Summary updated