Abstract
PBADet is a streamlined, one-stage, anchor-free framework designed for the joint detection and association of human parts with their parent bodies. By reversing the traditional "body-to-part" association logic to a "part-to-body" center offset, PBADet achieves superior scalability and accuracy. Evaluated on datasets like BodyHands and COCOHumanParts, PBADet establishes new state-of-the-art benchmarks while maintaining high computational efficiency, making it highly suitable for real-time applications.
Introduction
In many computer vision tasks, detecting an object is only half the battle. For human-centric applications—ranging from gesture-controlled medical interfaces to action recognition in dense crowds—it is essential to associate specific parts, such as hands and faces, with the correct individual. This task, known as Part-Body Association (PBA), presents unique challenges in localization and relational mapping. Traditional methods often struggle with the complexity of these relationships, particularly as the number of parts or the density of the crowd increases.
The Limitations of Current Approaches
Existing PBA methods generally fall into two categories, both of which possess inherent flaws:
- Two-Stage Methods: These detect bodies and parts independently before employing a second neural network or heuristic rules to link them. These are often computationally expensive and fragile; if the initial detection stage fails, the association stage cannot recover.
- Body-to-Part One-Stage Methods: Methods like BPJDet detect a person and then predict offsets to all potential parts. This approach scales poorly; as the number of part types increases (e.g., adding elbows, knees, or feet), the output layer complexity grows linearly. Furthermore, predicting offsets for occluded or off-screen parts wastes significant computational resources on "dead end" predictions.
PBADet: The Part-to-Body Paradigm
PBADet addresses these inefficiencies through an anchor-free, one-stage architecture centered on a "Part-to-Body" association principle. Instead of the body searching for its parts, each detected part predicts a single 2D offset pointing to the center of its parent body.
This "flip" in logic creates a robust one-to-one correspondence. While a body may have zero, one, or two hands visible, a visible hand always belongs to exactly one person. This makes the ground-truth labels for training significantly more consistent and the mathematical representation more stable.
Technical Innovations
- Concise Representation: By focusing on the part-to-body vector, PBADet reduces the mathematical complexity of the output. For a system tracking three parts (head, left hand, right hand), PBADet utilizes a vector of length 10, compared to the length 15 required by previous state-of-the-art models.
- Task-Aligned Learning: Borrowing from the Task-aligned One-stage Object Detection (TOOD) framework, the model uses a learning strategy that ensures the parts with the best localization also have the highest classification scores, leading to cleaner detections.
- Multi-Scale Prediction: The model operates across different feature map scales to handle people and parts of varying sizes, from a tiny hand in the distance to a large face in the foreground.
Experimental Results
The researchers validated PBADet across a wide range of human and animal datasets, including BodyHands, COCOHumanParts, CrowdHuman, and AnimalPose.
PBADet outperformed existing models across the board:
- BodyHands: Achieved a Hand AP of 89.1, significantly higher than the previous best of 84.8. It also reached a Joint AP of 85.98, proving its superiority in both detection and association.
- COCOHumanParts: The model delivered state-of-the-art performance, particularly excelling in "subordinate" metrics that measure how accurately parts are linked to bodies.
- Efficiency: Despite its increased accuracy, the YOLOv7-based version of PBADet is smaller and faster than many competitors, making it viable for real-time deployment.
Discussion and Future Work
PBADet demonstrates that scalability in part-body association is best achieved through simplicity. By using a part-to-body offset, the model's complexity remains constant regardless of how many parts are added to the detection list.
While the model excels in most scenarios, extremely crowded scenes where body centers overlap perfectly remain a challenge for distance-based matching. Future iterations may move beyond bounding boxes to integrate with pose estimation keypoints or segmentation masks for even more granular association. Additionally, the framework's success with animal datasets suggests it could be applied to any "parent-child" relationship in computer vision, such as associating wheels with vehicles or trailers with trucks.
Summary updated