Decoupling Classification and Localization in Object Detection

This direction is merged into another direction: task-specific feature decoupling. Its last version stays readable and citable here.

Version 1 of 127 papers28 conceptsfrozen 2026-10-04

In modern object detection, a single neural network must answer two very different questions: What is this object? and Where exactly are its boundaries? For years, the standard approach was to use a "sibling head" architecture, where a shared feature representation is fed into two parallel branches—one for classification and one for bounding box regression.

However, researchers have identified a fundamental "spatial conflict" inherent in this design. Classification and localization have diametrically opposed feature requirements. Forcing them to share the same spatial sampling points creates a tug-of-war that limits the performance of even the most powerful detectors.

The Fundamental Conflict of Interests

The core of the problem lies in what makes a feature "good" for a specific task.

  1. Classification thrives on translation-invariant and semantic features. To identify a "cat," the network looks for salient internal cues like ears or whiskers. These features are often concentrated at the object's center. As Qin Ran and Ran Qin note in MRDet, classification also benefits from features that are invariant to rotation.
  2. Localization (Regression) requires translation-aware and geometric features. To draw a perfect box, the network needs to know exactly where the edges and corners are. These features are located at the object's boundaries, far from the semantic center.

When a detector uses a shared feature grid—as is common in one-stage detectors—it tries to predict both category and coordinates from the same pixel or anchor. Yan Xu and Yang Li argue in PDNet that this creates a spatial misalignment: the point with the highest classification confidence is rarely the point that provides the most accurate boundary information.

Task-aware Spatial Disentanglement

To resolve this, researchers have moved toward "decoupling" the feature extraction process. Instead of forcing tasks to share a single spatial point, they allow each task to "look" where it needs to.

Guanglu Song and Yu Liu introduced Task-aware Spatial Disentanglement (TSD) in Revisiting the Sibling Head in Object Detector. TSD breaks the rigid link between the classification and localization branches. It allows the classification head to adaptively sample features from the object's interior while the localization head independently seeks out boundary-specific information. This simple shift—letting each task choose its own optimal spatial sampling—yields significant accuracy gains without changing the underlying backbone.

Similarly, Shiyi Lan and Zhou Ren proposed SaccadeNet, which mimics human eye movements. It first predicts an object center (for classification) and then "saccades" to informative corner keypoints to refine the boundaries. This explicit separation ensures that semantic classification isn't degraded by noisy boundary features.

Aligning Confidence with Accuracy

Decoupling features solves the input problem, but there is also an output problem: misalignment between classification scores and localization quality. Often, a detector will assign a high confidence score to a bounding box that is actually poorly aligned with the object.

Chengjian Feng and Yujie Zhong addressed this in TOOD by introducing a Task-Aligned Head. Instead of treating the two branches as independent silos, TOOD uses a learning strategy that explicitly encourages the classification head to be most confident at the spatial locations where the regression head is most accurate.

Another approach to this alignment is Mutual Supervision (MuSu), proposed by Ziteng Gao and Limin Wang 0002 in Mutual Supervision for Dense Object Detection. MuSu breaks the tradition of using the same training samples for both tasks. Instead, it allows the classification head to supervise which samples the regression head should focus on, and vice versa. This cross-task communication ensures that the final detections are both highly confident and precisely localized.

Beyond Standard Detection

The necessity of decoupling extends into more complex domains:

  • Human-Object Interaction (HOI): Desen Zhou and Jian Wang show in DisTR that identifying an object's box requires looking at its extremities, while identifying an action requires looking at the "interaction zone" between the human and the object.
  • Open-Vocabulary Detection: Zitong Li and Jinzhuo Wu in OSSD demonstrate that Vision-Language Models like CLIP are excellent at semantic recognition but poor at spatial localization. By decoupling these stages, they can leverage CLIP's knowledge without sacrificing box precision.
  • Multispectral Detection: In DPDETR, Junjie Guo and Chenqiang Gao decouple position detection to handle misalignments between infrared and visible light sensors, ensuring that the "where" from one modality doesn't confuse the "what" from another.

By acknowledging that classification and localization are distinct cognitive tasks with unique spatial needs, these researchers have moved the field away from the "one-size-fits-all" shared head toward more specialized, task-aware architectures.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_decoupling_classification_and_localizati_fa33601d,
  title = {Decoupling Classification and Localization in Object Detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decoupling-classification-and-localization-in-object-detection/rdr_fa33601dce3e4312a8f434d7f8ec2073},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_decoupling_classification_and_localizati_fa33601d_v1,
  title = {Decoupling Classification and Localization in Object Detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decoupling-classification-and-localization-in-object-detection/rdr_fa33601dce3e4312a8f434d7f8ec2073/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}