Abstract
This paper presents a novel real-time tracking system capable of improving body pose estimation algorithms in distributed camera networks. The first stage of our approach introduces a linear Kalman filter operating at the body joints level, used to fuse single-view body poses coming from different detection nodes of the network and to ensure temporal consistency between them. The second stage, instead, refines the Kalman filter estimates by fitting a hierarchical model of the human body having constrained link sizes in order to ensure the physical consistency of the tracking. The effectiveness of the proposed approach is demonstrated through a broad experimental validation, performed on a set of sequences whose ground truth references are generated by a commercial marker-based motion capture system. The obtained results show how the proposed system outperforms the considered state-of-the-art approaches, granting accurate and reliable estimates. Moreover, the developed methodology constrains neither the number of persons to track, nor the number, position, synchronization, frame-rate, and manufacturer of the RGB-D cameras used. Finally, the real-time performances of the system are of paramount importance for a large number of real-world applications.
Introduction
Human Body Pose Estimation (HBPE) is a cornerstone of computer vision, essential for applications ranging from human-robot collaboration to clinical gait analysis. However, real-time multi-person tracking remains challenging. Single-camera setups often suffer from self-occlusion, while multi-camera networks frequently struggle with hardware synchronization, varying frame rates, and temporal instability—often resulting in "jittery" skeletons where limb lengths appear to change frame-to-frame. This paper introduces a robust, real-time system designed to work with any number of heterogeneous, unsynchronized RGB-D cameras to produce stable, physically consistent 3D skeletons.
Methodology
The system follows a tracking-by-detection architecture divided into two primary stages:
1. Temporal Fusion via Kalman Filtering
Each camera node detects 2D/3D poses independently using a CNN. A central node receives these detections as a stream of updates. Because the cameras are not synchronized, the system utilizes a linear Kalman filter that treats every incoming detection as a timestamped update. This filter tracks both the position and velocity of every joint. To handle sensor noise, it employs an adaptive noise scheme: the filter's "trust" (measurement noise variance) is scaled based on the detector's confidence score. Additionally, an outlier rejection mechanism filters out detections that imply physically impossible joint displacements.
2. Physical Consistency via Hierarchical Optimization
Standard detectors often produce "rubbery" skeletons. To correct this, the system fits the Kalman-filtered joints to a hierarchical human body model. Using the Ceres Solver, the system minimizes an energy function that penalizes deviations from the subject's actual limb lengths and original joint orientations. By calculating average limb lengths in initial frames and enforcing them as constraints, the system ensures the 3D reconstruction remains physically realistic.
Experimental Validation
Due to the lack of public multi-camera RGB-D datasets with high-quality ground truth, the authors developed a custom setup using four Kinect v2 cameras. Ground truth was established using a professional-grade BTS SMART-DX motion capture system with 12 cameras and passive reflective markers. The validation sequences included both static and dynamic movements of up to two subjects.
Results and Discussion
The proposed system was compared against state-of-the-art methods, including OpenPose and OpenPTrack. As shown in the performance data, the proposed solution significantly improves accuracy, achieving mean joint errors between 5.5 cm and 9.5 cm, whereas OpenPose averaged between 8.3 cm and 11.3 cm on the same sequences.
In terms of speed, the system maintains real-time performance on standard consumer hardware (i7 CPU, GTX 1060 GPU), tracking a single person at ~34 fps and four people at ~19 fps. Furthermore, the results demonstrate that adding more cameras to the network consistently reduces error and standard deviation, proving the fusion algorithm effectively leverages multiple viewpoints.
Conclusion
This work demonstrates a scalable and hardware-agnostic approach to 3D human tracking. By treating asynchronicity as a stream of updates and enforcing physical limb constraints, the system provides a stable foundation for real-world HBPE applications. Future work will explore deeper integration of the hierarchical model into the initial detection phase to further reduce latency and improve performance in crowded environments.
Summary updated