TL;DR
Visual Recognition by Request (ViRReq) is a new segmentation paradigm that breaks down complex, fine-grained visual recognition into a series of on-demand, text-guided "requests." By decoupling recognition granularity from annotation completeness, it successfully trains on highly incomplete hierarchical datasets and seamlessly adapts to new concepts via few-shot incremental learning.
What Problem Does This Paper Address?
Traditional computer vision models try to predict a fixed set of labels all at once. However, human vision operates at an unlimited granularity—we can look at a person, then focus on their head, then their eyes, or even their eyelashes.
Forcing deep learning models to label everything at the finest granularity creates a fundamental conflict between granularity and certainty. As visual details get finer, human annotations naturally become sparse, noisy, and highly incomplete. Standard models struggle to learn from these partial labels and find it difficult to adapt when new categories are introduced later.
Methods
The authors introduce ViRReq, a framework that reframes image segmentation as an iterative, query-based process guided by a hierarchical, text-based knowledge base (a tree dictionary of objects and parts). Instead of outputting a flat label map, the system answers two atomic types of requests sequentially:
- Type-I Requests (Whole-to-Part Semantic Segmentation): Given an object instance, the model segments it into its constituent parts (e.g., breaking a "car" down into "wheels," "doors," and "windows"). This is achieved by using text embeddings from a pre-trained, frozen CLIP model as queries against a SegFormer backbone.
- Type-II Requests (Instance Segmentation): Given a semantic region, the model isolates individual instances based on a user-defined or densely sampled "probing pixel" using a modified CondInst framework.
To ensure structural integrity, Top-Down Filtering is applied so that part predictions strictly reside within their parent object boundaries. Furthermore, a novel Data Versioning mechanism binds training images to specific versions of the knowledge base. This allows the model to learn new concepts incrementally without treating missing labels in older data as negative errors.
Datasets
The framework was evaluated on two major segmentation datasets featuring whole-part hierarchies:
- Cityscapes-Panoptic-Parts (CPP): Extends the Cityscapes dataset with part-level annotations for vehicles and humans.
- ADE20K: A highly complex dataset with multi-level whole-part hierarchies (objects, parts, and parts-of-parts). It is notorious for its sparse and incomplete annotations, making it a prime testbed for ViRReq.
Key Insights and Ideas
- On-Demand Recognition Handles Sparse Data: By performing recognition only when requested, the model bypasses the need for a perfectly complete dataset. It accepts that a small object might not have part labels, while a larger instance of the same class does.
- Language as a Flexible Class Index: Replacing fixed integer IDs with CLIP text embeddings allows the system to perform open-domain recognition. It can detect anomalous/unseen classes and transfer part knowledge compositionally (e.g., applying "car wheel" knowledge to a "bus" or "van" without new training data).
- Mask Sampling for Robust Probing: Standard interactive segmentation models expect clicks near the center of an object. ViRReq introduces mask sampling during training, making the model highly robust to probes placed anywhere on an instance, even near boundaries.
Key Results
- Competitive Performance on CPP: ViRReq achieved competitive accuracy against specialized architectures, scoring 62.5% PartPQ using a SegFormer-B5 backbone, with clear advantages in recognizing object parts.
- First Part-Aware Results on ADE20K: The authors are the first to report quantitative part-aware segmentation results on ADE20K. They introduced a new metric, Hierarchical Panoptic Quality (HPQ), to evaluate multi-level prediction trees. ViRReq outperformed standard multi-task baselines (33.9% vs 31.8% HPQ on all classes).
- Strong Few-Shot Incremental Learning: ViRReq successfully absorbed 50 new object classes and 19 part classes using only 20 labeled examples per concept, leaving the base classes mostly unaffected.
Limitations and Future Work
The current framework relies on a manually predefined knowledge base structure. The authors note that designing an automatic method to learn and update this hierarchical dictionary directly from data is a critical next step. Additionally, future work could optimize the prompt engineering side to close the gap between upstream vision-language pre-training and downstream request-based fine-tuning.
Summary updated