MEOW: Medical Open-World Interactive Segmentation

Philip Matthias Winter, Maria Wimmer, Bernhard Schäfl, David Major, Dimitrios Lenis, Astrid Berg, Theresa Neubauer, Ariharasudhan Muthusami, Johannes Novotny, Georg Zotti, Katja BühlerarXiv 2026 · arXiv

Medical imaging is a "wild west" for AI. A model trained to segment a liver in a standard CT scan might completely choke when it encounters a rare tumor, a different imaging modality like MRI, or even just a scan from a different hospital. Traditionally, fixing this meant collecting new data and retraining the model—a slow, expensive process that often leads to "catastrophic forgetting" of old knowledge.

In their paper "MEOW: Medical Open-World Interactive Segmentation," Winter et al. introduce a framework that allows models to learn new structures interactively and instantly, without ever updating their underlying neural network weights.

MEOW framework overview: (a) Flexible interactive properties, (b) Segmenting unknown structures and learning, (c) Improving performance via learned structures.

TL;DR

MEOW is the first framework for Open-World Interactive Segmentation (OW-IS) that turns frozen, non-interactive backbones into adaptive learners. By decoupling feature extraction from a dynamic memory module, it enables models to segment and "remember" novel anatomical structures in milliseconds, achieving state-of-the-art performance in open-set scenarios.

What Problem Does This Paper Address?

Most medical segmentation models operate in a "closed-world": they only know the classes they were trained on. When they encounter something new (an "unknown-unknown"), they either fail silently or provide overconfident, incorrect predictions.

While Interactive Segmentation (IS) allows users to guide models with clicks or brushes, most IS systems are still "closed-set"—they don't get smarter after the interaction ends. If the model sees the same unknown structure again, the user has to start from scratch. Existing methods for "lifelong learning" usually rely on gradient-based fine-tuning, which is computationally heavy, slow, and risks corrupting the model's original capabilities.

Methods

MEOW (Medical Open-World) solves this by using a modular, parameter-free transduction approach. The system consists of three main parts:

  1. Frozen Feature Extractor ($F$): A standard pre-trained backbone (like a U-Net) that stays completely frozen. It just provides the "eyes" for the system.
  2. Memory Module ($M$): A database that stores "exemplars"—feature pyramids of previously segmented structures.
  3. Transduction Module ($G$): This is the "brain." Instead of a learned decoder, it uses transductive reasoning. When a user provides a click, the system performs a k-nearest neighbor (k-NN) search across the current image and the memory module to propagate labels.
The MEOW data flow: Features are extracted once, then the transduction module uses user guidance and memory to generate predictions.

The "Open-World" loop happens when a user accepts a segmentation. That result is instantly "consolidated" into memory. The next time the model sees a similar structure, it retrieves that knowledge to provide a high-quality initial guess, reducing the number of clicks the user needs.

To ensure the results are spatially smooth, the authors implemented Hierarchical Message Passing (HMP), which acts as a powerful post-processing step to maintain consistency across different scales of the image.

Datasets

The authors tested MEOW's versatility across a wide range of domains:

  • Human CT: TotalSegmentator, MSD (Liver, Pancreas, Spleen), and KiTS23 (Kidney).
  • Micro-CT ($\mu$CT): Mouse organ datasets and InsectAnatomy (ant brains), representing a massive domain shift from human anatomy.
  • MRI: MSD BRATS (brain tumors) to test cross-modality generalization.

Key Insights and Ideas

  • Decoupling Perception from Knowledge: The core "aha!" moment is that you don't need to retrain a neural network to teach it a new concept. If the features are rich enough, you just need a way to map those features to new labels.
  • Learning in Milliseconds: Because "learning" in MEOW is just a memory write operation, it takes about 7.6 ms. Compare this to traditional fine-tuning, which takes minutes or hours.
  • Zero-Shot to Open-World: A model trained only on human abdominal CT scans can successfully segment an ant brain or an MRI tumor simply by leveraging its own high-quality predictions stored in memory.
  • User-Centric Control: The framework includes interactive hyperparameters (via sliders) that let clinicians adjust things like "message passing strength" or "classification threshold" in real-time to refine results.
The interactive UI allows users to provide clicks and adjust hyperparameters via sliders to refine the segmentation.

Key Results

  • Efficiency: MEOW consistently improves its own predictions. With as few as 8 stored samples, the model showed significant performance gains on completely unseen classes.
  • Superior Open-Set Performance: In tests on unknown $\mu$CT data, MEOW outperformed specialized interactive foundation models like VISTA3D and SAM-Med3D.
  • Robustness: The system is "backbone-agnostic." It worked effectively with multiple different frozen feature extractors, even showing decent results with a completely untrained (randomized) backbone, proving the power of the transductive k-NN approach.
  • Real-Time Interaction: Inference speeds are well under 1 second, making it viable for clinical use.
Qualitative results showing MEOW handling diverse tasks like segmenting mouse lungs, ant brains, and human kidneys with minimal user interaction.

Limitations and Future Work

  • Memory Footprint: Storing high-dimensional feature pyramids in memory can be resource-intensive, especially for large 3D volumes.
  • Architecture Constraints: The current implementation requires "hourglass-shaped" CNN architectures (like U-Nets) to build its hierarchical graphs; it doesn't yet support Transformer-based backbones.
  • Future Directions: The authors suggest exploring self-supervised feature extractors and applying MEOW to longitudinal studies (tracking changes in a patient over time) where the model can "remember" the patient's specific anatomy from previous scans.

Summary updated

Abstract

In open-world recognition, machine learning models not only make predictions on unknown structures ("generalized open-set"), but also continually learn and improve from those predictions over time. Interactive segmentation, despite its access to user feedback, has so far remained confined to the open-set setting. In this work we introduce MEOW, the first framework that turns non-interactive segmentation backbones into interactive open-world learners without updating learnable model parameters. MEOW combines parameter-free transductive reasoning with memory consolidation to accumulate knowledge and thereby improve future performance. Focusing on medical data, we show that MEOW consistently improves its own predictions on classes never seen during training, and transfers zero-shot to unfamiliar domains, including micro-CT data and other imaging modalities.
BibTeX
@misc{winter2026meow,
  author = {Philip Matthias Winter and Maria Wimmer and Bernhard Schäfl and David Major and Dimitrios Lenis and Astrid Berg and Theresa Neubauer and Ariharasudhan Muthusami and Johannes Novotny and Georg Zotti and Katja Bühler},
  title = {{MEOW: Medical Open-World Interactive Segmentation}},
  year = {2026},
  eprint = {2611.13935},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13935},
}
Figure