In computer vision, the "holy grail" for tasks like person search and multi-object tracking is a single, end-to-end neural network that can do everything at once. Ideally, a model should look at a crowded street and simultaneously locate every person (detection) and assign them a unique digital fingerprint (re-identification, or Re-ID).
However, researchers have discovered a fundamental "optimization conflict" at the heart of this goal. As Bharti Munjal, Fabio Galasso, and Sikandar Amin observe in Knowledge Distillation for End-to-End Person Search, detection and identification are inherently contradictory tasks. Detection requires the model to generalize—to find the "commonness" that makes all humans look like humans. Re-ID requires the model to specialize—to find the "uniqueness" that distinguishes one specific individual from everyone else.
The Geometry of Contradiction
When we force a single backbone to learn both tasks, the gradients literally pull the model parameters in opposing directions. This is not just a theoretical nuisance; it leads to "negative transfer," where the pursuit of better identification accuracy actively degrades the detector's precision.
Chao Liang and colleagues describe this in Rethinking the competition between detection and ReID in Multi-Object Tracking as a fundamental trade-off. Detection thrives on invariance: a person should be detected regardless of their clothes, lighting, or pose. Re-ID thrives on sensitivity: it needs to capture the exact texture of a jacket or the specific geometry of a face.
This conflict extends beyond people. In SwinFace, Lixiong Qin and Mei Wang note that identity recognition must be invariant to facial expressions, while expression detection requires high sensitivity to those same muscle movements. Similarly, Francisco Massa found that detection requires viewpoint invariance, while pose estimation requires viewpoint sensitivity.
Decoupling Through Architecture
To solve this, researchers have moved away from "naive" joint learning toward architectures that explicitly decouple these competing signals.
One approach is the Two-Stream model. In Person Search via a Mask-Guided Two-Stream CNN Model, Di Chen and Shanshan Zhang demonstrate that using independent streams for the foreground person and the original image patch prevents the features from becoming "inexpressive descriptors" that fail at both tasks.
Other researchers use Reciprocal Networks or specialized attention. CSTrack uses a network to disentangle these tasks, while OneTrack by Jiawei He and Zhaoxiang Zhang identifies and coordinates contradictory classification gradients to ensure they don't cancel each other out during training.
Mathematical Disentanglement: Norms and Directions
Perhaps the most elegant solution to this conflict is Norm-Aware Embedding (NAE). Di Chen and Bernt Schiele proposed a simple geometric split:
- Magnitude (Norm): Used for detection. A high-norm vector indicates "this is a person."
- Direction (Angle): Used for identity. The vector's orientation in hyperspace represents the unique individual.
By separating the "what" (norm) from the "who" (angle), a single feature vector can serve both masters without compromise.
Dynamic Routing and Task Consistency
The latest frontier involves making the network itself "aware" of which task it is performing at any given moment. Minsu Kim and Kwanghoon Sohn introduced Mixture of Submodules (MoS), which dynamically routes data through specialized sub-sections of the network based on the task and domain. This allows the model to switch "modes" between generalizing for detection and specializing for identification.
Furthermore, Bingpeng Ma and Shiguang Shan argue in TCTS that the conflict isn't just in the features, but in the consistency of the pipeline. They suggest that detectors must be "query-aware," learning to prioritize the specific types of features that the subsequent Re-ID model will actually use, rather than just finding any generic pedestrian.
By acknowledging that "finding" and "identifying" are fundamentally different cognitive acts, these researchers have transformed person search from a compromised multi-task struggle into a coordinated, hierarchical system.