Improving Real-Time Omnidirectional 3D Multi-Person Human Pose Estimation with People Matching and Unsupervised 2D-3D Lifting

Pawel Knap, Peter Hardy, Alberto Tamajo, Hwasup Lim, Hansung Kim

Abstract

This paper presents a real-time system for 3D multi-person pose estimation using a single 360° omnidirectional camera fused with mmWave radar sensors. By leveraging unsupervised 2D-3D lifting and a novel cross-modal matching algorithm, the system effectively resolves depth ambiguities inherent in monocular vision. The approach demonstrates robust performance in challenging outdoor lighting conditions where traditional infrared sensors fail, maintaining a consistent frame rate suitable for real-time applications.

Introduction

Estimating 3D human poses from a single camera is a fundamentally ill-posed problem due to depth and scale ambiguities. Determining whether a subject is small and near or large and distant is mathematically complex without auxiliary depth information. While LiDAR and RGB-D sensors (e.g., Kinect) offer depth data, they are often cost-prohibitive or perform poorly in direct sunlight due to infrared interference. This research proposes a cost-effective, scalable solution combining 360° vision with mmWave radar to achieve accurate, real-time 3D localization and pose estimation in diverse environments.

Methodology

The proposed system utilizes a hardware-software fusion architecture designed for efficiency and robustness.

Hardware Configuration

The setup consists of a Ricoh Theta V omnidirectional camera, providing a full 360° field of view, and three TI AWR1843BOOST mmWave radar sensors. This configuration ensures comprehensive spatial coverage and immunity to ambient lighting conditions.

Processing Pipeline

The software pipeline follows a multi-stage approach as illustrated in the system architecture:

  1. 2D Keypoint Detection: OpenPose extracts 2D skeletal keypoints from the panoramic imagery.
  2. Unsupervised 2D-3D Lifting: The LInKs neural network converts 2D keypoints into 3D coordinates. Unlike supervised models, LInKs predicts depth offsets relative to a root joint (the pelvis), enhancing generalization to novel poses.
  3. Radar-Vision Matching: A binary search tree-based algorithm associates 2D camera poses with 3D radar detections. This process uses the disparity between image x-coordinates and radar-detected spatial coordinates to resolve depth.
  4. Calibration: The coordinate systems are aligned using the Levenberg-Marquardt algorithm to calculate affine transformations for the radar sensors.
System architecture illustrating the flow from sensor input to 3D visualization.

Experimental Setup

The system was evaluated using both standard benchmarks and custom real-world data. The LInKs network was retrained on the Human3.6M dataset to align with the OpenPose keypoint format. Real-time performance was tested on a consumer-grade laptop equipped with an NVIDIA RTX 3060 GPU to assess matching accuracy and localization error in live scenarios.

Results and Discussion

The fusion of radar and vision provides a "depth anchor" that solves the scale ambiguity of the lifting process. The radar provides absolute distance, which scales the relative 3D pose generated by the neural network.

Performance Metrics

  • Real-Time Capability: The system achieves a frame rate of 7-8 fps, which remains nearly constant regardless of the number of people in the scene.
  • Accuracy Improvements: The refined matching algorithm yielded a 4.63% increase in accuracy over previous versions.
  • Localization: Calibration drastically reduced radar localization error; for instance, error in "Radar 1" decreased from 23.89% to 2.52%.
  • Robustness: The system handles self-occlusion well, with a Mean Per Joint Position Error (MPJPE) of 52.1mm even when a limb is hidden.
Visual results showing 2D pose detection on panoramic images (top) and the corresponding 3D reconstructed poses (bottom).

Conclusion

This research demonstrates that combining omnidirectional vision with mmWave radar offers a viable path for real-time 3D pose estimation in uncontrolled environments. While challenges remain regarding "void spaces" between radar coverage areas and total torso occlusions, the current system provides a robust foundation for future work in scalable human-computer interaction and surveillance.

Summary updated

BibTeX
@inproceedings{knap2024improving,
  author = {Pawel Knap and Peter Hardy and Alberto Tamajo and Hwasup Lim and Hansung Kim},
  title = {{Improving Real-Time Omnidirectional 3D Multi-Person Human Pose Estimation with People Matching and Unsupervised 2D-3D Lifting}},
  booktitle = {ICEIC},
  year = {2024},
  doi = {10.1109/iceic61013.2024.10457094},
}
Figure