Visual Recognition by Request

Chufeng Tang, Lingxi Xie, Xiaopeng Zhang, Xiaolin Hu, Qi Tian

TL;DR

Visual Recognition by Request (ViRReq) is a new segmentation paradigm that breaks down complex, fine-grained visual recognition into a series of on-demand, text-guided "requests." By decoupling recognition granularity from annotation completeness, it successfully trains on highly incomplete hierarchical datasets and seamlessly adapts to new concepts via few-shot incremental learning.

What Problem Does This Paper Address?

Traditional computer vision models try to predict a fixed set of labels all at once. However, human vision operates at an unlimited granularity—we can look at a person, then focus on their head, then their eyes, or even their eyelashes.

Forcing deep learning models to label everything at the finest granularity creates a fundamental conflict between granularity and certainty. As visual details get finer, human annotations naturally become sparse, noisy, and highly incomplete. Standard models struggle to learn from these partial labels and find it difficult to adapt when new categories are introduced later.

Figure 1. An illustration of unlimited granularity in visual recog- nition. Top: an example image from ADE20K [52] with instance- level and part-level annotations. Middle: more and more incom- plete annotations of instances, parts, and parts of parts. Bottom: as granularity goes finer, higher uncertainty occurs in recogniz- ing the boundary (left) and semantic class (right) of objects and/or parts – here, green, blue and red texts indicate labeled, unlabeled (but defined), and unlabeled (and undefined) objects, respectively.

Methods

The authors introduce ViRReq, a framework that reframes image segmentation as an iterative, query-based process guided by a hierarchical, text-based knowledge base (a tree dictionary of objects and parts). Instead of outputting a flat label map, the system answers two atomic types of requests sequentially:

  • Type-I Requests (Whole-to-Part Semantic Segmentation): Given an object instance, the model segments it into its constituent parts (e.g., breaking a "car" down into "wheels," "doors," and "windows"). This is achieved by using text embeddings from a pre-trained, frozen CLIP model as queries against a SegFormer backbone.
  • Type-II Requests (Instance Segmentation): Given a semantic region, the model isolates individual instances based on a user-defined or densely sampled "probing pixel" using a modified CondInst framework.

To ensure structural integrity, Top-Down Filtering is applied so that part predictions strictly reside within their parent object boundaries. Furthermore, a novel Data Versioning mechanism binds training images to specific versions of the knowledge base. This allows the model to learn new concepts incrementally without treating missing labels in older data as negative errors.

Figure 2. An illustration of the overall pipeline of ViRReq that is built upon a pre-defined knowledge base. Given an input image, ViRReq starts with an empty field (i.e., nothing is annotated or recognized) and iteratively executes Type-I/II requests towards recognition results of finer granularity. The sub-knowledge for Type-I requests is extracted from the knowledge base. The labeler can stop after any number of steps, yet the model can learn from incomplete annotations without difficulties. To save space, we combine a few Type-II requests into one step. We only display five steps here, while a typical annotation/recognition procedure often include much more steps (and requests).

Datasets

The framework was evaluated on two major segmentation datasets featuring whole-part hierarchies:

  1. Cityscapes-Panoptic-Parts (CPP): Extends the Cityscapes dataset with part-level annotations for vehicles and humans.
  2. ADE20K: A highly complex dataset with multi-level whole-part hierarchies (objects, parts, and parts-of-parts). It is notorious for its sparse and incomplete annotations, making it a prime testbed for ViRReq.

Key Insights and Ideas

  • On-Demand Recognition Handles Sparse Data: By performing recognition only when requested, the model bypasses the need for a perfectly complete dataset. It accepts that a small object might not have part labels, while a larger instance of the same class does.
  • Language as a Flexible Class Index: Replacing fixed integer IDs with CLIP text embeddings allows the system to perform open-domain recognition. It can detect anomalous/unseen classes and transfer part knowledge compositionally (e.g., applying "car wheel" knowledge to a "bus" or "van" without new training data).
  • Mask Sampling for Robust Probing: Standard interactive segmentation models expect clicks near the center of an object. ViRReq introduces mask sampling during training, making the model highly robust to probes placed anywhere on an instance, even near boundaries.
Type-I Request Processing

Key Results

  • Competitive Performance on CPP: ViRReq achieved competitive accuracy against specialized architectures, scoring 62.5% PartPQ using a SegFormer-B5 backbone, with clear advantages in recognizing object parts.
  • First Part-Aware Results on ADE20K: The authors are the first to report quantitative part-aware segmentation results on ADE20K. They introduced a new metric, Hierarchical Panoptic Quality (HPQ), to evaluate multi-level prediction trees. ViRReq outperformed standard multi-task baselines (33.9% vs 31.8% HPQ on all classes).
  • Strong Few-Shot Incremental Learning: ViRReq successfully absorbed 50 new object classes and 19 part classes using only 20 labeled examples per concept, leaving the base classes mostly unaffected.

Limitations and Future Work

The current framework relies on a manually predefined knowledge base structure. The authors note that designing an automatic method to learn and update this hierarchical dictionary directly from data is a critical next step. Additionally, future work could optimize the prompt engineering side to close the gap between upstream vision-language pre-training and downstream request-based fine-tuning.

Summary updated

Abstract

Humans have the ability of recognizing visual semantics in an unlimited granularity, but existing visual recognition algorithms cannot achieve this goal. In this paper, we establish a new paradigm named visual recognition by request (ViRReq) to bridge the gap. The key lies in decomposing visual recognition into atomic tasks named requests and leveraging a knowledge base, a hierarchical and text-based dictionary, to assist task definition. ViRReq allows for (i) learning complicated whole-part hierarchies from highly incomplete annotations and (ii) inserting new concepts with minimal efforts. We also establish a solid baseline by integrating language-driven recognition into recent semantic and instance segmentation methods, and demonstrate its flexible recognition ability on CPP and ADE20K, two datasets with hierarchical whole-part annotations.
BibTeX
@inproceedings{tang2022visual,
  author = {Chufeng Tang and Lingxi Xie and Xiaopeng Zhang and Xiaolin Hu and Qi Tian},
  title = {{Visual Recognition by Request}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
}
Figure