OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association

Sven Kreiss, Lorenzo Bertoni, Alexandre Alahi

Abstract

Many image-based perception tasks can be formulated as detecting, associating and tracking semantic keypoints, e.g., human body pose estimation and tracking. In this work, we present a general framework that jointly detects and forms spatio-temporal keypoint associations in a single stage, making this the first real-time pose detection and tracking algorithm. We present a generic neural network architecture that uses Composite Fields to detect and construct a spatio-temporal pose which is a single, connected graph whose nodes are the semantic keypoints (e.g., a person's body joints) in multiple frames. For the temporal associations, we introduce the Temporal Composite Association Field (TCAF) which requires an extended network architecture and training method beyond previous Composite Fields. Our experiments show competitive accuracy while being an order of magnitude faster on multiple publicly available datasets such as COCO, CrowdPose and the PoseTrack 2017 and 2018 datasets. We also show that our method generalizes to any class of semantic keypoints such as car and animal parts to provide a holistic perception framework that is well suited for urban mobility such as self-driving cars and delivery robots.

Introduction

For autonomous vehicles to navigate safely, they need "social intelligence"—the ability to read the body language of pedestrians and the intent of other drivers. Simply putting a bounding box around a person is insufficient; the car needs to know if a pedestrian is looking at the road or if a cyclist is about to signal a turn. While human pose estimation has improved, tracking those poses across video frames remains a challenge. Most existing methods are "top-down" (detect person, then find joints) or multi-stage (detect joints, then link them across frames), which are often computationally expensive and fragile in crowded urban environments.

OpenPifPaf provides a holistic perception framework for humans, animals, and vehicles in urban scenes.

Methodology

The authors propose a bottom-up approach based on Composite Fields. Instead of looking for whole people, the network predicts fine-grained fields that describe the "what" and "where" of body parts.

Composite Fields

The framework utilizes three primary field types:

  1. Composite Intensity Fields (CIF): These detect the location, confidence, and size of individual keypoints (e.g., a left shoulder).
  2. Composite Association Fields (CAF): These predict the connections between keypoints within a single frame (e.g., connecting a shoulder to an elbow).
  3. Temporal Composite Association Fields (TCAF): The core innovation of this work, TCAF predicts associations between keypoints across different frames, linking a keypoint at time $t$ to the same keypoint at time $t-1$ directly within the neural network's output.
Visualization of (a) Composite Intensity Fields, (b) Composite Association Fields, and (c) Temporal Composite Association Fields.

Decoding and Robustness

To turn these fields into actual poses, a Greedy Decoder with a Frontier is used. This priority queue builds poses by starting from the most confident keypoints and following the association fields. To handle occlusions, the authors use Denser Pose Skeletons, adding redundant connections (e.g., nose-to-shoulder) to ensure a pose remains connected even if a single joint is hidden.

Experiments and Results

The model was evaluated on diverse benchmarks including COCO, CrowdPose, PoseTrack, ApolloCar3D, and Animal-Pose.

  • Speed: OpenPifPaf achieved 13.7 FPS on the CrowdPose dataset, roughly an order of magnitude faster than previous top-performing methods.
  • Accuracy: On the COCO test-dev set, it achieved a competitive 71.9 AP, outperforming other bottom-up methods like OpenPose.
  • Tracking: On PoseTrack 2018, it reached a 61.7 MOTA at 12.2 FPS, demonstrating high accuracy at real-time speeds.
  • Generalization: The framework achieved an 86.1% detection rate for vehicle keypoints on ApolloCar3D, outperforming specialized vehicle models.

Conclusion

OpenPifPaf provides a high-performance, single-stage framework for joint detection and tracking. By predicting temporal associations alongside spatial ones, it avoids the overhead of separate tracking algorithms. While performance on animals is currently lower due to dataset size limitations, the framework's ability to generalize across humans, cars, and animals suggests a path toward a unified perception engine for autonomous systems. Future work will explore shared backbone architectures to handle multiple classes simultaneously.

Summary updated

Abstract

Many image-based perception tasks can be formulated as detecting, associating and tracking semantic keypoints, e.g., human body pose estimation and tracking. In this work, we present a general framework that jointly detects and forms spatio-temporal keypoint associations in a single stage, making this the first real-time pose detection and tracking algorithm. We present a generic neural network architecture that uses Composite Fields to detect and construct a spatio-temporal pose which is a single, connected graph whose nodes are the semantic keypoints (e.g., a person's body joints) in multiple frames. For the temporal associations, we introduce the Temporal Composite Association Field (TCAF) which requires an extended network architecture and training method beyond previous Composite Fields. Our experiments show competitive accuracy while being an order of magnitude faster on multiple publicly available datasets such as COCO, CrowdPose and the PoseTrack 2017 and 2018 datasets. We also show that our method generalizes to any class of semantic keypoints such as car and animal parts to provide a holistic perception framework that is well suited for urban mobility such as self-driving cars and delivery robots.
BibTeX
@article{kreiss2022openpifpaf,
  author = {Sven Kreiss and Lorenzo Bertoni and Alexandre Alahi},
  title = {{OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association}},
  journal = {IEEE Trans. Intell. Transp. Syst.},
  year = {2022},
  url = {https://openreview.net/forum?id=ooUT1T2JKE},
}
Figure