A Robust Filter for Marker-less Multi-person Tracking in Human-Robot Interaction Scenarios

Enrico Martini, Harshil Parekh, Shaoting Peng, Nicola Bombieri, Nadia Figueroa

Introduction

Safe and efficient human-robot interaction (HRI) in collaborative environments requires precise and reliable human tracking. While marker-based motion capture systems provide high accuracy, their intrusive nature and high cost limit their applicability in real-world settings. Marker-less tracking using Human Pose Estimation (HPE) models offers a more flexible alternative; however, these systems are prone to noise, depth inaccuracies, and tracking loss during occlusions. Such instabilities result in "robot jitter," which significantly undermines human trust and perceived safety. This paper presents a novel filtering pipeline designed to mitigate these issues by integrating biomechanical constraints and a dynamic permanence filter.

Methodology

The proposed system is a "plug-and-play" pipeline that processes raw HPE data before it reaches the robot controller. It consists of three primary modules:

  1. Spatial Evaluation: This module applies biomechanical heuristics to filter out physically impossible detections. By estimating subject height and verifying limb proportions (e.g., forearm length relative to height), the system identifies and penalizes "hallucinated" joints.
  2. Temporal Tracking: To maintain identity across frames, the system employs the Hungarian method. The cost matrix is uniquely constructed by combining Euclidean distance with prediction confidence, ensuring robust label assignment even during person-to-person intersections.
  3. Confidence-Based Permanence Filter: At the core of the pipeline is an Object Permanence Filter (OPF) implemented via a particle filter. Unlike static filters, this module dynamically adjusts its noise parameters based on the HPE confidence score. During occlusions, the system transitions to a dynamics module that predicts human movement based on prior trajectories, maintaining smooth interaction until the subject is re-acquired.
Overview of the proposed filtering pipeline, illustrating the spatial, temporal, and permanence nodes.

Experimental Evaluation

The pipeline was validated using an Intel RealSense D455 RGB-D camera and a Franka Emika Panda robot arm. Seven participants performed four collaborative tasks (T0-T3) designed to test various challenges, including non-linear motion and heavy occlusions (e.g., box handovers). Ground truth data was collected using an OptiTrack marker-based system to quantify tracking error.

Results and Discussion

The proposed filter demonstrated significant improvements over baseline HPE data and standard Kalman filter implementations. Quantitative analysis revealed a 49% reduction in Mean Absolute Error (MAE) and a 71% reduction in the standard deviation (STD) of the error. This reduction in STD is particularly critical, as it directly translates to the elimination of erratic robot movements.

Comparison of 2D motion projections. The proposed filter (d) closely aligns with the green ground truth, avoiding the erratic jumps seen in the baseline (a) and Kalman variants (b, c).

Beyond technical metrics, the system significantly enhanced the user experience. Subjective surveys indicated that perceived safety and comfort scores rose from 2.2/5 with standard filtering to 4.6/5 with the proposed pipeline. Furthermore, the filter maintains real-time performance with a negligible latency of approximately 8ms, making it suitable for 30Hz robotic applications.

Conclusion

This research introduces a robust, real-time filtering pipeline that bridges the gap between noisy marker-less HPE and the high-precision requirements of HRI. By leveraging biomechanical common sense and dynamic confidence-based filtering, the system ensures smooth robot behavior even under challenging conditions. Future work will explore the integration of multi-camera setups and advanced intent prediction to further enhance tracking persistence during prolonged occlusions.

Summary updated

Abstract

Pursuing natural and marker-less human-robot interaction (HRI) has been a long-standing robotics research focus, driven by the vision of seamless collaboration without physical markers. Marker-less approaches promise an improved user experience, but state-of-the-art struggles with the challenges posed by intrinsic errors in human pose estimation (HPE) and depth cameras. These errors can lead to issues such as robot jittering, which can significantly impact the trust users have in collaborative systems. We propose a filtering pipeline that refines incomplete 3D human poses from an HPE backbone and a single RGB-D camera to address these challenges, solving for occlusions that can degrade the interaction. Experimental results show that using the proposed filter leads to more consistent and noise-free motion representation, reducing unexpected robot movements and enabling smoother interaction.
BibTeX
@inproceedings{martini2024robust,
  author = {Enrico Martini and Harshil Parekh and Shaoting Peng and Nicola Bombieri and Nadia Figueroa},
  title = {{A Robust Filter for Marker-less Multi-person Tracking in Human-Robot Interaction Scenarios}},
  booktitle = {RO-MAN},
  year = {2024},
  url = {https://openreview.net/forum?id=yb3qs5E4ky},
}
Figure