Abstract
This paper presents a real-time system for 3D multi-person pose estimation using a single 360° omnidirectional camera fused with mmWave radar sensors. By leveraging unsupervised 2D-3D lifting and a novel cross-modal matching algorithm, the system effectively resolves depth ambiguities inherent in monocular vision. The approach demonstrates robust performance in challenging outdoor lighting conditions where traditional infrared sensors fail, maintaining a consistent frame rate suitable for real-time applications.
Introduction
Estimating 3D human poses from a single camera is a fundamentally ill-posed problem due to depth and scale ambiguities. Determining whether a subject is small and near or large and distant is mathematically complex without auxiliary depth information. While LiDAR and RGB-D sensors (e.g., Kinect) offer depth data, they are often cost-prohibitive or perform poorly in direct sunlight due to infrared interference. This research proposes a cost-effective, scalable solution combining 360° vision with mmWave radar to achieve accurate, real-time 3D localization and pose estimation in diverse environments.
Methodology
The proposed system utilizes a hardware-software fusion architecture designed for efficiency and robustness.
Hardware Configuration
The setup consists of a Ricoh Theta V omnidirectional camera, providing a full 360° field of view, and three TI AWR1843BOOST mmWave radar sensors. This configuration ensures comprehensive spatial coverage and immunity to ambient lighting conditions.
Processing Pipeline
The software pipeline follows a multi-stage approach as illustrated in the system architecture:
- 2D Keypoint Detection: OpenPose extracts 2D skeletal keypoints from the panoramic imagery.
- Unsupervised 2D-3D Lifting: The LInKs neural network converts 2D keypoints into 3D coordinates. Unlike supervised models, LInKs predicts depth offsets relative to a root joint (the pelvis), enhancing generalization to novel poses.
- Radar-Vision Matching: A binary search tree-based algorithm associates 2D camera poses with 3D radar detections. This process uses the disparity between image x-coordinates and radar-detected spatial coordinates to resolve depth.
- Calibration: The coordinate systems are aligned using the Levenberg-Marquardt algorithm to calculate affine transformations for the radar sensors.
Experimental Setup
The system was evaluated using both standard benchmarks and custom real-world data. The LInKs network was retrained on the Human3.6M dataset to align with the OpenPose keypoint format. Real-time performance was tested on a consumer-grade laptop equipped with an NVIDIA RTX 3060 GPU to assess matching accuracy and localization error in live scenarios.
Results and Discussion
The fusion of radar and vision provides a "depth anchor" that solves the scale ambiguity of the lifting process. The radar provides absolute distance, which scales the relative 3D pose generated by the neural network.
Performance Metrics
- Real-Time Capability: The system achieves a frame rate of 7-8 fps, which remains nearly constant regardless of the number of people in the scene.
- Accuracy Improvements: The refined matching algorithm yielded a 4.63% increase in accuracy over previous versions.
- Localization: Calibration drastically reduced radar localization error; for instance, error in "Radar 1" decreased from 23.89% to 2.52%.
- Robustness: The system handles self-occlusion well, with a Mean Per Joint Position Error (MPJPE) of 52.1mm even when a limb is hidden.
Conclusion
This research demonstrates that combining omnidirectional vision with mmWave radar offers a viable path for real-time 3D pose estimation in uncontrolled environments. While challenges remain regarding "void spaces" between radar coverage areas and total torso occlusions, the current system provides a robust foundation for future work in scalable human-computer interaction and surveillance.
Summary updated