SDR-GAIN: A High Real-Time Occluded Pedestrian Pose Completion Method for Autonomous Driving

Honghao Fu, Yongli Gu, Yidong Yan, Yilang Shen, Yiwen Wu, Libo SunIEEE Trans. ITS 2026 · DOI

TL;DR

This paper introduces SDR-GAIN, a novel real-time framework for accurately reconstructing occluded pedestrian poses in autonomous driving scenarios. By directly learning human pose from keypoint coordinates and employing a self-supervised adversarial approach with specialized generators for head and torso, SDR-GAIN achieves significant improvements in accuracy and maintains microsecond-level inference speeds, crucial for real-time applications.

What Problem Does This Paper Address?

Autonomous driving systems rely heavily on understanding pedestrian behavior for safety. A critical component of this is pedestrian pose estimation, which helps infer intentions. However, in real-world driving, pedestrians are frequently occluded by vehicles, vegetation, or other objects. Traditional pose estimation methods struggle to accurately reconstruct these occluded keypoints, leading to a loss of crucial information and potentially compromising the robustness of autonomous systems. While existing methods try to address occlusion, they often suffer from high computational costs and inference latency, making them unsuitable for time-critical autonomous driving applications. This paper tackles the challenge of achieving both accurate and real-time occluded pedestrian pose completion.

Methods

The core of SDR-GAIN is a self-supervised generative adversarial network (GAN) framework designed for pose completion. Instead of relying on visual cues to detect occlusion, SDR-GAIN directly learns from the numerical distribution of keypoint coordinates.

Here's a breakdown of their approach:

  • Initial Pose Estimation: The process begins with an off-the-shelf pose estimation model (OpenPose) to extract initial keypoints from input images or video frames.
  • Pose Standardization: To improve learning, the extracted keypoints undergo several standardization steps:
  • Separation: Head and torso keypoints are treated separately. This is because their distributions differ, and training a single model for both can hinder learning.
  • Rotation: Keypoints are rotated to a consistent angle using reference points (ears for the head, shoulders for the torso) to standardize posture tilt variations.
  • Dimensionality Reduction: Keypoint coordinates are projected onto the x and y axes and normalized to a [0, 1] range. This creates a uniform data distribution.
  • Generative Adversarial Imputation: Two specialized GAN generators are trained: one for the head and one for the torso. These generators take the standardized, potentially incomplete keypoint data and a "hint mask" (indicating missing keypoints) as input. They learn to impute the missing coordinates.
  • Masked Learning: The GANs are trained using a masked learning strategy. A "mask vector" indicates which keypoints are missing, and a "hint vector" guides the generator.
  • Residual Structures: The generators and discriminators incorporate residual structures to facilitate deeper learning and mitigate vanishing gradient issues.
  • Reverse Processing: After the generators impute the missing keypoints, the process is reversed (un-rotation, un-projection) to restore the complete pedestrian pose in its original coordinate space.

Datasets

The authors trained and evaluated their SDR-GAIN framework on two widely used datasets:

  • COCO: A large-scale dataset containing over 100,000 human bodies with keypoint annotations.
  • JAAD: A dataset specifically designed to capture the behavioral variability of traffic participants, making it highly relevant for autonomous driving scenarios.

Key Insights and Ideas

  • Direct Keypoint Learning: The most significant insight is moving away from visual occlusion detection and directly learning from the numerical distribution of keypoint coordinates. This simplifies the model and potentially reduces computational overhead.
  • Head/Torso Separation: Treating head and torso keypoints separately is crucial. Their distinct spatial distributions require specialized handling for effective imputation.
  • Standardization is Key: The combination of rotation and dimensionality reduction significantly standardizes the pose data, making it easier for the GANs to learn and generalize.
  • Microsecond-Level Inference: Achieving such low inference times for pose completion is a major breakthrough, directly addressing the real-time demands of autonomous driving.
  • GANs for Imputation: Leveraging GANs, particularly with the "hint mask" mechanism, proves effective for imputing missing spatial data like keypoints.

Key Results

  • Accuracy: SDR-GAIN achieved at least a 47.4% reduction in RMSE compared to other baseline imputation algorithms on both COCO and JAAD datasets.
  • Real-time Performance: The method achieves microsecond-level inference time, making it highly efficient for real-time applications. This is a significant advantage over Transformer-based methods that offer comparable accuracy but at much higher latencies.
  • Qualitative Results: Visual comparisons (Fig. 6) demonstrate SDR-GAIN's ability to accurately restore missing keypoints in various occlusion scenarios, outperforming other baselines.
  • Ablation Studies: Experiments confirmed the effectiveness of separation, dimensionality reduction, and the multi-generator strategy. They also highlighted the importance of choosing appropriate loss functions (Huber loss for head, Huber + residual for torso) and model structures based on data characteristics.

Limitations and Future Work

  • Dataset Scale: The authors acknowledge that the limited scale of complete pedestrian pose data available for GAN training can constrain performance. They suggest that utilizing larger or domain-specific datasets could further enhance SDR-GAIN's effectiveness.
  • Adversarial Training Instability: Like many GANs, SDR-GAIN can face convergence difficulties such as vanishing gradients, mode collapse, and oscillations. They mention regularization techniques as a potential solution.
  • Future Work: While not explicitly detailed, the paper implies that further exploration of regularization techniques to stabilize adversarial training and the use of larger, more diverse datasets are promising avenues for future research. The current work focuses on a single-frame approach, and extending it to temporal sequences could also be a future direction.

Summary updated

BibTeX
@article{fu2026sdr,
  author = {Honghao Fu and Yongli Gu and Yidong Yan and Yilang Shen and Yiwen Wu and Libo Sun},
  title = {{SDR-GAIN: A High Real-Time Occluded Pedestrian Pose Completion Method for Autonomous Driving}},
  journal = {IEEE Trans. Intell. Transp. Syst.},
  year = {2026},
  doi = {10.1109/tits.2025.3630759},
}
Figure