Real-Time Multi-Person Pose Tracking using Data Assimilation

Caterina Buizza, Tobias Fischer, Yiannis Demiris

Introduction

While deep learning has significantly advanced the field of pose estimation—identifying joint locations in static images—the transition to robust pose tracking across video sequences remains a significant challenge. Most contemporary models treat video frames as independent problems, leading to high computational costs, identity inconsistency (where a subject's ID flips between frames), and physically impossible joint configurations, such as swapped limbs or flickering detections during brief occlusions.

Researchers from Imperial College London have addressed these issues by introducing a framework based on Data Assimilation, a technique traditionally used in weather forecasting. By integrating Kalman filters with deep learning pose estimators, they have developed a system capable of real-time multi-person tracking that adheres to physical laws of motion.

Methodology

The proposed framework is modular, designed to enhance existing pose estimators like OpenPose or LightTrack. The core innovation lies in the use of Kalman Filters to bridge the gap between expensive neural network inferences.

The Skip Strategy and Kinematic Modeling

To achieve real-time performance, the system employs a "skip" strategy: the heavy neural network runs only every $N$ frames. For the intervening frames, the system utilizes a Kalman filter to predict joint positions. The researchers found that an Acceleration Model was most effective for human motion, as it accounts for the non-constant velocities of extremities like wrists and ankles.

Figure 2: Detail of joint tracking and limb consistency maintained by the framework.

Velocity and ID Matching

To refine the Kalman filter's predictions, the framework uses Lucas-Kanade optical flow to estimate joint velocity between frames. Furthermore, a smart identification module scores skeleton pairs based on proximity and scale, ensuring that individual identities remain consistent throughout the video sequence.

Experimental Setup

The framework was evaluated using the PoseTrack 2018 dataset, a benchmark for multi-person pose tracking in unconstrained environments. The training set was used to calibrate "noise matrices," teaching the Kalman filter the degree of uncertainty associated with specific neural network outputs, while the validation set served for performance benchmarking.

Key Results and Discussion

The integration of Data Assimilation provided several key advantages:

  • Real-Time Efficiency: The framework boosted models like OpenPose and LightTrack to over 30 FPS. By adjusting the tracking window ($N$), users can tune the balance between processing speed and raw accuracy.
  • Consistency Improvements: OpenPose, originally a static image analyzer, saw a dramatic improvement in identity consistency, effectively transforming it into a robust video tracker.
  • Physical Realism: By combining spatial data from machine learning with temporal reasoning from Data Assimilation, the system successfully filters out "noisy" detections that defy human kinematics.
Figure 1: Multi-frame tracking results demonstrating identity consistency and joint stability across a sequence.

While the system is highly effective, it faces limitations regarding "drift" if the tracking window $N$ becomes too large (typically 10+ frames), particularly during sudden, erratic movements. Additionally, new individuals entering the frame during a tracking phase may not be detected until the next neural network inference cycle.

Conclusion

This research demonstrates that combining the spatial strengths of deep learning with the temporal reasoning of Data Assimilation creates a powerful tool for real-time computer vision. Future work may explore Extended Kalman Filters (EKF) or neural-network-based motion models to better handle the complex, non-linear nature of human movement.

Summary updated

Abstract

We propose a framework for the integration of data assimilation and machine learning methods in human pose estimation, with the aim of enabling any pose estimation method to be run in real-time, whilst also increasing consistency and accuracy. Data assimilation and machine learning are complementary methods: the former allows us to make use of information about the underlying dynamics of a system but lacks the flexibility of a data-based model, which we can instead obtain with the latter. Our framework presents a real-time tracking module for any single or multi-person pose estimation system. Specifically, tracking is performed by a number of Kalman filters initiated for each new person appearing in a motion sequence. This permits tracking of multiple skeletons and reduces the frequency that computationally expensive pose estimation has to be run, enabling online pose tracking. The module tracks for N frames while the pose estimates are calculated for frame N + 1 . This also results in increased consistency of person identification and reduced inaccuracies due to missing joint locations and inversion of left-and right-side joints.
BibTeX
@inproceedings{buizza2020real,
  author = {Caterina Buizza and Tobias Fischer and Yiannis Demiris},
  title = {{Real-Time Multi-Person Pose Tracking using Data Assimilation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  doi = {10.1109/wacv45572.2020.9093442},
}
Figure