Employing Shadows for Multi-Person Tracking Based on a Single RGB-D Camera

Wei Gai, Meng Qi, Mingcong Ma, Lu Wang, Chenglei Yang, Juan Liu, Yulong Bian, Gerard de Melo, Shijun Liu, Xiangxu Meng

TL;DR

This paper introduces a novel approach to multi-person tracking using a single RGB-D camera by leveraging human shadows as a cue. By fusing shadow data with skeleton tracking, the method effectively handles long-duration and full-body occlusions, a significant challenge for existing systems.

What Problem Does This Paper Address?

Tracking multiple people in real-time, especially in crowded or dynamic environments, is a persistent challenge. A key difficulty arises from occlusions, where one person blocks the view of another, leading to a loss of tracking data. While RGB-D cameras like Kinect offer depth information alongside color, they still struggle with complete or prolonged occlusions when using a single camera. Existing solutions often rely on multiple cameras (increasing cost and complexity) or sophisticated tracking-by-detection methods that can be sensitive to detection errors and partial occlusions. This paper aims to address the limitations of single-camera tracking by utilizing an often-overlooked cue: human shadows.

Methods

The core of the proposed method is a Shadow-based Tracking Algorithm (STA) that complements standard skeleton tracking. When a person's skeleton data is available (i.e., no occlusion), the system uses it to determine their position. However, when occlusion occurs and skeleton data is lost, the system switches to using the person's shadow.

The process involves:

  1. Shadow Detection: Shadows are identified in the RGB image by comparing it to a background model. Pixels with lower luminosity than the background are considered potential shadow pixels.
  2. Shadow Region Identification: To efficiently find the shadow, the system first identifies the likely location of the shadow based on the position of the non-occluded person's head (obtained from skeleton data) and a pre-determined parameter indicating whether the shadow is to the left or right of the person. This narrows down the search area.
  3. Shadow Extraction: Within the identified region, the system extracts the shadow pixels.
  4. Position Estimation via Shadow: The extracted shadow contour is then used to estimate the occluded person's position. This involves fitting a line to the shadow contour and mapping it to the 3D skeleton space. The intersection of this shadow line with a projected line from the camera's origin to the ground plane (derived from the non-occluded person's position) provides an estimate of the occluded person's 3D position.

This approach is integrated into a tracking framework that continuously attempts to obtain skeleton data and falls back to shadow-based tracking when skeleton data is lost.

Datasets

The paper does not mention specific datasets used for training. The evaluation was conducted using experiments with human participants in a controlled indoor environment using two Kinect sensors. One Kinect was used for tracking (k1), and another (k2) was used as a ground truth reference for the occluded person's trajectory.

Key Insights and Ideas

  • Shadows as a Valuable Tracking Cue: The fundamental insight is that shadows, often dismissed as noise, can provide crucial positional information for tracking, especially during occlusions.
  • Complementary Tracking: The method effectively fuses two distinct sources of information: direct skeleton data when available and indirect shadow data when skeleton data is lost. This fusion makes the tracking system more robust.
  • Leveraging Existing Hardware: The approach relies solely on a single RGB-D camera (like Kinect), avoiding the need for multiple sensors or external equipment, making it more practical and cost-effective.
  • Handling Long-Duration Occlusions: The shadow-based approach shows promise in maintaining tracking even during prolonged periods where a person is completely out of view.

Key Results

  • Accuracy: The method demonstrated a mean tracking accuracy of 0.8. The tracking deviation along the X and Z axes was found to be relatively small, with average deviations of 0.17 and 0.16 respectively.
  • Robustness to Occlusion: Experiments showed that the algorithm could effectively compute the position of people even when they were completely occluded for extended periods.
  • Real-time Performance: The algorithm achieved an average computation time of approximately 15 frames per second on a 2.8GHz Intel Core i5 processor, indicating its feasibility for real-time applications on modest hardware.
  • Versatility: The method proved effective for various motion patterns, including movement along single axes, combined axes, and arbitrary directions, as well as transitions between stationary and moving states.

Limitations and Future Work

  • Shadow Detection Sensitivity: The success of the method hinges on accurate shadow detection. If shadows are too faint (e.g., due to strong ambient light outdoors or very bright indoor lighting), they might not be captured accurately.
  • Outdoor Environments: The method is primarily robust in indoor settings. Outdoor tracking without sunlight might be challenging.
  • Complex Shadow Scenarios: The current work assumes a single shadow per person. Scenarios with multiple overlapping shadows (e.g., from multiple light sources or multiple occluding people) are not fully addressed.
  • Future Work: The authors suggest integrating Region of Interest (ROI) segmentation and Frustum PointNets for more accurate multi-user tracking. They also propose extending the method to support more than two people and developing more elaborate shadow tracking mechanisms for complex shadow interactions. The method could also be extended to other RGB-D cameras and video cameras.

Summary updated

Abstract

Although there are many algorithms to track people that are walking, existing methods mostly fail to cope with occluded bodies in the setting of multi-person tracking with one camera. In this paper, we propose a method to use people’s shadows as a clue to track them instead of treating shadows as mere noise. We introduce a novel method to track multiple people by fusing shadow data from the RGB image with skeleton data, both of which are captured by a single RGB Depth (RGB-D) camera. Skeletal tracking provides the positions of people that can be captured directly, while their shadows are used to track them when they are no longer visible. Our experiments confirm that this method can efficiently handle full occlusions. It thus has substantial value in resolving the occlusion problem in multi-person tracking, even with other kinds of cameras.
BibTeX
@inproceedings{gai2020employing,
  author = {Wei Gai and Meng Qi and Mingcong Ma and Lu Wang and Chenglei Yang and Juan Liu and Yulong Bian and Gerard de Melo and Shijun Liu and Xiangxu Meng},
  title = {{Employing Shadows for Multi-Person Tracking Based on a Single RGB-D Camera}},
  booktitle = {Sensors},
  year = {2020},
  url = {https://openreview.net/forum?id=KWls73D4SlS},
}
Figure