OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association

Sven Kreiss, Lorenzo Bertoni, Alexandre Alahi

TL;DR

OpenPifPaf is a high-performance, bottom-up framework that jointly detects and tracks semantic keypoints (like human joints or vehicle parts) in a single stage. By introducing Temporal Composite Association Fields (TCAF), it achieves state-of-the-art accuracy while running an order of magnitude faster than previous multi-stage methods, making it viable for real-time applications like self-driving cars.

What Problem Does This Paper Address?

For autonomous vehicles to navigate safely, they need "social intelligence"—the ability to read the body language of pedestrians and the intent of other drivers. Simply putting a bounding box around a person isn't enough; the car needs to know if a pedestrian is looking at the road or if a cyclist is about to signal a turn.

While human pose estimation has improved, tracking those poses across video frames remains a massive challenge. Most existing methods are "top-down" (detect person, then find joints) or multi-stage (detect joints, then link them across frames). These are often computationally expensive, slow, and fragile in crowded urban environments where people frequently overlap or disappear behind obstacles (occlusions).

Methods

The authors propose a bottom-up approach based on Composite Fields. Instead of looking for whole people, the network predicts fine-grained fields that describe the "what" and "where" of body parts:

  1. Composite Intensity Fields (CIF): These detect the location, confidence, and size of individual keypoints (e.g., a left shoulder).
  2. Composite Association Fields (CAF): These predict the connections between keypoints within a single frame (e.g., connecting a shoulder to an elbow).
  3. Temporal Composite Association Fields (TCAF): This is the paper's core innovation. TCAF predicts associations between keypoints across different frames. This allows the system to link a "left knee" at time $t$ to the same "left knee" at time $t-1$ directly within the neural network's output.

To turn these fields into actual poses, they use a Greedy Decoder with a Frontier. This is essentially a priority queue that builds poses by starting from the most confident keypoints and following the association fields. They also use Denser Pose Skeletons, adding redundant connections (like nose-to-shoulder) to ensure a pose doesn't "break" if a single joint (like an ear) is hidden.

Datasets

The model was trained and evaluated on a wide variety of benchmarks:

  • Human Pose: COCO (standard benchmark), CrowdPose (focuses on overlapping people), and PoseTrack 2017/2018 (video tracking).
  • Vehicles: ApolloCar3D and KITTI (detecting car "joints" like lights and wheels).
  • Animals: Animal-Pose (dogs, cats, cows, horses, sheep).

Key Insights and Ideas

  • Single-Stage is Key: By predicting temporal associations (TCAF) alongside spatial ones, the model avoids the overhead of a separate tracking algorithm. This is why it achieves such high speeds.
  • Redundancy for Robustness: Standard pose models use a "sparse" tree-like skeleton. OpenPifPaf uses a "dense" skeleton. If a person's arm is blocked by a pole, the model can still link the hand to the torso through alternative, longer-range connections.
  • Holistic Perception: The framework isn't just for humans. The authors demonstrate that the same architecture can track car parts or animal joints, suggesting a path toward a unified "perception engine" for robots.
  • Handling Occlusion: The "Greedy Decoder" is specifically designed to handle "self-hidden" keypoints, preventing the model from getting confused when one limb crosses over another.

Key Results

  • Speed: OpenPifPaf is significantly faster than its predecessors. On the CrowdPose dataset, it achieved 13.7 FPS, roughly an order of magnitude faster than previous top-performing methods.
  • Accuracy: On the COCO test-dev set, it achieved a competitive 71.9 AP (Average Precision), outperforming other bottom-up methods like OpenPose and PersonLab.
  • Tracking: On PoseTrack 2018, it reached a 61.7 MOTA (Multi-Object Tracking Accuracy) at 12.2 FPS, proving it can maintain high accuracy even at high speeds.
  • Generalization: The model achieved an 86.1% detection rate for vehicle keypoints on ApolloCar3D, outperforming specialized vehicle models.

Limitations and Future Work

While the model is highly efficient, the authors note that performance on animals is lower than on humans (47.8 AP), primarily due to the much smaller size of animal datasets (only 4,000 training instances).

Future work aims to investigate shared backbone architectures. Currently, separate heads are used for different classes; a single, unified backbone that handles humans, cars, and animals simultaneously would be the "holy grail" for a holistic autonomous vehicle perception system. They also suggest that incorporating 3D information could further improve reasoning in complex urban scenes.

Summary updated

Abstract

Many image-based perception tasks can be formulated as detecting, associating and tracking semantic keypoints, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> , human body pose estimation and tracking. In this work, we present a general framework that jointly detects and forms spatio-temporal keypoint associations in a single stage, making this the first real-time pose detection and tracking algorithm. We present a generic neural network architecture that uses Composite Fields to detect and construct a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">spatio-temporal pose</i> which is a single, connected graph whose nodes are the semantic keypoints ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g</i> ., a person’s body joints) in multiple frames. For the temporal associations, we introduce the Temporal Composite Association Field (TCAF) which requires an extended network architecture and training method beyond previous Composite Fields. Our experiments show competitive accuracy while being an order of magnitude faster on multiple publicly available datasets such as COCO, CrowdPose and the PoseTrack 2017 and 2018 datasets. We also show that our method generalizes to any class of semantic keypoints such as car and animal parts to provide a holistic perception framework that is well suited for urban mobility such as self-driving cars and delivery robots.
BibTeX
@article{kreiss2022openpifpaf,
  author = {Sven Kreiss and Lorenzo Bertoni and Alexandre Alahi},
  title = {{OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association}},
  journal = {IEEE Trans. Intell. Transp. Syst.},
  year = {2022},
  url = {https://openreview.net/forum?id=ooUT1T2JKE},
}

Loading related content...

Figure