Class Agnostic Localization for Open World Object Detection

766 concepts445 papers4 proposals

Standard object detectors suffer from a form of "tunnel vision." When a model is trained to optimize bounding boxes alongside a closed-set classification loss (such as cross-entropy), its region proposal mechanism becomes highly specialized to the visual features of only those training classes. Consequently, the detector learns to treat any novel, unseen, or unlabelled object simply as background clutter.

To break free from this closed-world assumption, a growing body of research focuses on class-agnostic localization. The core insight is simple yet powerful: objectness is a semantic-free visual property. An object's physical boundaries—its edges, contrast, symmetry, and spatial closure—can be recognized independently of what the object actually is. By mathematically and structurally decoupling "where" an object is from "what" it is, models can successfully propose and localize completely novel entities in open-world environments.

The Core Insight: Objectness is Semantic-Free

Why does class-agnostic localization work? In traditional architectures, the network's representation of an object is deeply entangled with its category label. If a network is never trained on the class "manatee," its region proposal network (RPN) will likely suppress a region containing a manatee because it lacks the features of a "car," "person," or "dog."

However, as early as the work by Brody Huval et al. in Deep learning for class-generic object detection, evidence showed that neural networks can learn a generalized representation of object boundaries that transfers seamlessly to novel categories. A bounding box around an unknown object shares the same geometric and structural grammar as a box around a known one. When we force a localization module to focus strictly on these class-independent properties, it develops a universal detector capable of flagging any distinct physical entity.

Architectural Decoupling: Separating Where from What

Modern open-world and open-vocabulary detectors operationalize this insight by structurally isolating the localization task.

Class-Agnostic Proposal Heads

Instead of predicting category-specific bounding boxes, the primary detection or region proposal heads are stripped of semantic labels. For example, Chengjian Feng et al. in PromptDet: Towards Open-Vocabulary Detection Using Uncurated Images and Weicheng Kuo et al. in Open-Vocabulary Object Detection upon Frozen Vision and Language Models employ class-agnostic box and mask prediction heads. By focusing the initial stages purely on drawing accurate boundaries, the model generates high-recall proposals for unseen objects, which are only later passed to downstream modules (like frozen vision-language models) for semantic alignment.

Eliminating the Classification Head

To completely eliminate the risk of known categories suppressing novel objects, some frameworks remove the classification branch entirely during discovery. Qian Zhang et al. demonstrate in OpenInst: A Simple Query-Based Method for Open-World Instance Segmentation that explicit classification branches are not strictly necessary for object discovery. The localization task itself inherently teaches learnable queries to distinguish objects from the background, preventing the model from filtering out structures that do not align with known training categories.

Traditional Coupled Pipeline:
[Image] ---> [Shared Features] ---> [Class-Specific RPN] ---> [Suppresses Unknowns]

Decoupled Class-Agnostic Pipeline:
[Image] ---> [Shared Features] ---> [Class-Agnostic Localization] ---> [High-Recall Boxes] ---> [Open-Vocabulary Classifier]

Enforcing Feature Orthogonality

Even when architectures separate the prediction heads, the underlying feature representations can remain implicitly biased toward known classes. If the backbone network only extracts features relevant to distinguishing a cat from a dog, the class-agnostic head will still fail on a novel object.

To solve this, Zhicheng Sun et al. in Exploring Orthogonality in Open World Object Detection propose enforcing mathematical independence between localization and classification features. The authors map the network's latent space into a polar coordinate system:

$$\text{Feature Space} \rightarrow (\text{Magnitude}, \text{Direction})$$

In this space, the vector magnitude is trained to represent generic objectness (localization), while the vector direction encodes semantic category (classification). Because magnitude and direction are orthogonal, the model's ability to locate an object becomes entirely insensitive to whether it recognizes the object's class, drastically increasing sensitivity to unknown categories.

Rethinking Objectives: Geometric and Foreground-Agnostic Losses

Training a truly class-agnostic localizer requires moving away from standard classification losses that penalize unlabelled objects.

One approach is to substitute semantic classification with pure geometric regression during the proposal phase. Juexiao Feng in Debiased Novel Category Discovering and Localization replaces standard classification loss with Intersection-over-Union (IoU) regression within the region proposal network. This forces the RPN to judge its success based on spatial overlap with any physical entity rather than semantic alignment.

Another strategy involves altering how proposals are matched during training. Yanghao Wang et al. introduce a framework in Random Boxes Are Open-world Object Detectors that utilizes a foreground-agnostic matching score. Instead of evaluating whether a proposed box matches a specific target label, the score aggregates the likelihood of a proposal belonging to any foreground class. This prevents the systematic suppression of high-confidence regions that contain real, albeit unlabelled, objects.

Broad Impact Across Vision Tasks

Decoupling localization from classification has unlocked significant capabilities across a variety of computer vision domains where predefined labels are unavailable or impractical:

By treating the discovery of "where" an object is as a universal, geometric constant, class-agnostic localization provides the foundation for vision systems to navigate, track, and reason about the infinite variety of the open world.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 469 papers
  2. Version 1 · 2026-10-04 · 613 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_class_agnostic_region_proposal_985dbb0d,
  title = {class-agnostic region proposal},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/class-agnostic-region-proposal/rdr_985dbb0de0634977930637a4a0315215},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_class_agnostic_region_proposal_985dbb0d_v2,
  title = {class-agnostic region proposal},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/class-agnostic-region-proposal/rdr_985dbb0de0634977930637a4a0315215/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake