PairDETR : Joint Detection and Association of Human Bodies and Faces

Ammar Ali, Georgii Gaikov, Denis Rybalchenko, Alexander Chigorin, Ivan Laptev, Sergey ZagoruykoCVPR 2024 · OpenReview

Abstract

Image and video analysis requires not only accurate object detection but also the understanding of relationships among detected objects. Common solutions to relation modeling typically resort to stand-alone object detectors followed by non-differentiable post-processing techniques. Recently introduced detection transformers (DETR) perform end-to-end object detection based on a bipartite matching loss. Such methods, however, lack the ability to jointly detect objects and resolve object associations. In this paper, we build on the DETR approach and extend it to the joint detection of objects and their relationships by introducing an approximated bipartite matching. While our method can generalize to an arbitrary number of objects, we here focus on the modeling of object pairs and their relations. In particular, we apply our method PairDETR to the problem of detecting human bodies and faces, and associating them for the same person. Our approach not only eliminates the need for hand-designed post-processing but also achieves excellent results for body-face associations. We evaluate PairDETR on the challenging CrowdHuman and CityPersons datasets and demonstrate a large improvement over the state of the art.

Introduction

In modern computer vision, detecting an object is often only the first step; understanding the relationships between detected entities is frequently the more critical task. A quintessential example is the association of a human face with a corresponding body. This capability is foundational for applications ranging from fitness tracking and gesture recognition to retail "virtual try-on" systems.

Traditionally, this association has been handled via a two-stage pipeline. First, separate detectors are deployed for bodies and faces. Second, heuristic-heavy post-processing—such as calculating bounding box overlaps or proximity—is used to link the detections. This approach is inherently fragile, particularly in crowded environments where bounding boxes frequently overlap. Furthermore, because these stages are decoupled, the detector cannot learn from its association errors during training. PairDETR addresses these issues by proposing an end-to-end transformer-based model that jointly detects and links human bodies and faces.

The PairDETR Framework

PairDETR is built upon the Deformable DETR architecture, extending it to treat detection and association as a unified graph prediction problem. The core shift is in the nature of the model's queries. Instead of a query predicting a single object, each query in PairDETR is designed to predict a pair—specifically, a body and its associated face.

The PairDETR architecture: Features are extracted and passed through a transformer to generate a set of pair predictions, which are then matched against ground truth using a specialized loss.

By using a single query to handle both parts of the pair, the model leverages the global context of a person's pose. This shared context allows the model to accurately locate small or blurry faces that a standalone face detector might miss, as it "knows" where the face should be relative to the detected body.

Approximated Bipartite Matching

The primary technical challenge in joint detection is the matching process during training. Standard DETR models use the Hungarian algorithm for bipartite matching between predicted and ground-truth boxes. However, matching pairs becomes mathematically complex (NP-hard) when components are missing—for instance, when a body is visible but the face is occluded or turned away.

To overcome this, the authors introduce an "approximated bipartite matching" strategy. The key innovation is the use of "dummy" nodes or "relatively estimated" classes.

The approximated matching strategy (c) simplifies the complex flow problem (b) by introducing dummy classes (yellow) to represent missing components in a pair, enabling standard bipartite matching.

By forcing the model to always predict a pair—even if one part is invisible—and assigning a specific label to these "hallucinated" parts, the authors transform a complex matching problem into a standard one. This allows the model to maintain the end-to-end differentiability of the DETR framework while learning spatial relationships directly from the data.

Experimental Results

The model was rigorously evaluated on two major pedestrian detection benchmarks: CrowdHuman, characterized by high density and occlusion, and CityPersons, focused on urban environments.

PairDETR demonstrated significant performance gains over existing state-of-the-art methods:

  • CrowdHuman: The model achieved a log-average miss-matching rate ($mMR^{-2}$) of 42%, representing an 8% absolute improvement over the previous leading method, BPJDet.
  • CityPersons: PairDETR outperformed the BFJ (Body-Face-Joint) baseline across all tested occlusion levels. Notably, in "heavy" occlusion scenarios, it reduced the miss-matching rate from 46.2% to 37.83%.

Crucially, the addition of the association task did not degrade raw detection performance. PairDETR maintained competitive Average Precision (AP) for both body and face detection, proving that joint learning can be mutually beneficial for both tasks.

Conclusion and Future Work

PairDETR successfully eliminates the need for ad-hoc post-processing heuristics by integrating object association directly into the transformer architecture. While the current implementation focuses on pairs, the "dummy node" strategy provides a roadmap for more complex relationship modeling.

Future research may address the exponential complexity that arises when scaling to larger groups of objects (e.g., body, face, hands, and tools). Additionally, exploring multi-reference point designs within the transformer could further refine the spatial accuracy of the predicted associations.

Summary updated

Abstract

Image and video analysis requires not only accurate object detection but also the understanding of relationships among detected objects. Common solutions to relation modeling typically resort to stand-alone object detectors followed by non-differentiable post-processing techniques. Re-cently introduced detection transformers (DETR) perform end-to-end object detection based on a bipartite matching loss. Such methods, however, lack the ability to jointly detect objects and resolve object associations. In this paper, we build on the DETR approach and extend it to the joint detection of objects and their relationships by introducing an approximated bipartite matching. While our method can generalize to an arbitrary number of objects, we here focus on the modeling of object pairs and their relations. In particular, we apply our method PairDETR to the problem of detecting human bodies and faces, and associating them for the same person. Our approach not only eliminates the need for hand-designed post-processing but also achieves excellent results for body-face associations. We evaluate PairDETR on the challenging CrowdHuman and CityPersons datasets and demonstrate a large improvement over the state of the art. Our training code and pretrained models are available at https://github.com/mts-ai/pairdetr
BibTeX
@inproceedings{ali2024pairdetr,
  author = {Ammar Ali and Georgii Gaikov and Denis Rybalchenko and Alexander Chigorin and Ivan Laptev and Sergey Zagoruyko},
  title = {{PairDETR : Joint Detection and Association of Human Bodies and Faces}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=zarU92JGuM},
}
Figure