TL;DR
This paper proposes a novel method to identify and track multiple people in a camera network by fusing data from CCTV cameras with inertial sensor data from their mobile phones. By treating this as a probabilistic data association problem, the system can uniquely identify individuals and overcome challenges like occlusions and sensing gaps, effectively solving the "correspondence problem" in computer vision.
What Problem Does This Paper Address?
The paper tackles the long-standing challenge of reliably identifying and tracking multiple individuals in a shared environment, particularly when using camera networks. While CCTV cameras can detect and localize people, they struggle with uniquely identifying them, especially when people cross paths, enter or leave the scene, or are occluded. This is known as the "correspondence problem" in computer vision, where the number of possible track combinations grows exponentially with time and the number of people, making it computationally intractable. Existing methods often require extensive training for identification (e.g., gait or facial recognition) or rely on specialized, often intrusive, hardware. The authors aim to leverage the ubiquity of mobile phones with their built-in inertial sensors to provide a more robust and privacy-preserving solution.
Methods
The core of the proposed method is a probabilistic data-association framework that fuses information from two sources:
- CCTV Cameras: Provide anonymous location detections of people within their field of view. These are termed "split measurements" because they lack explicit identity information.
- Mobile Phones: Provide inertial measurements (accelerometer and magnetometer) along with a unique phone ID. These measurements capture local motion but cannot inherently localize the user.
The system models this as a two-measurement Hidden Markov Model (HMM) for each person, where one measurement is from the camera (location) and the other from the phone (inertial data). The key innovation is to infer the statistical dependence between these two types of measurements to establish a link between a phone's ID and a camera's detection.
The problem is broken down into two parts:
- Instantaneous ID Assignments: For a single timestep, the system calculates a likelihood matrix (Ωk) representing the probability that a given camera detection (y) and inertial measurement (w) originated from the same person. This is achieved by marginalizing out the unknown true state (x) and then using a Hungarian assignment algorithm to find the optimal matching between detections and phone IDs.
- Maximum a Posteriori (MAP) Estimate: To improve robustness and handle temporal aspects, the system extends the instantaneous approach to a recursive MAP estimation over multiple timesteps. This involves computing the joint probability of the state sequence and observations, leveraging the Markov assumption. The system recursively updates its estimate, making it efficient for real-time applications.
A crucial aspect is the use of a "lazy tracker" approach to handle "proximity ambiguities" (when people are close together). Instead of making immediate decisions during such ambiguous periods, the system postpones the final track stitching and identification until later, significantly reducing the computational complexity of the optimization problem.
Datasets
The paper describes the creation of datasets for learning emission and transmission probabilities for the HMM. These were generated from experiments where people walked in the testbed, and their motion was recorded by both the camera system and inertial sensors.
- Emission and Transmission Probabilities: Heuristically estimated from experiments involving a person walking in the camera network for one minute. Figures 5 and 6 illustrate the learned probability distributions for motion and walking/stopped states.
- Simulation Datasets: Used to characterize performance under varying target densities and inter-ambiguity times. These simulations employed a random-waypoint model with emulated sensor noise.
- Experimental Datasets:
- 1-person traces: 15 experimental traces where one person walked for 1 minute, later superimposed to create multi-person scenarios with known ground truth.
- Ambiguity-resolution tests: 36 traces involving two people to test the system's ability to disambiguate after an ambiguity.
Key Insights and Ideas
- Sensor Fusion as a "Glue": The core idea is that inertial sensor data from mobile phones acts as a "glue" to connect anonymous camera detections with unique phone IDs, effectively solving the correspondence problem.
- Bypassing the Correspondence Problem: By directly inferring the statistical dependence between camera and phone measurements, the system sidesteps the computationally expensive task of explicitly matching tracks across cameras.
- Leveraging Ubiquitous Technology: The method capitalizes on the widespread availability of smartphones and existing CCTV infrastructure, making it practical for deployment in many environments.
- Privacy-Preserving Potential: The system can identify individuals without necessarily extracting identifying features like faces or gait, offering a more privacy-conscious approach.
- Lazy Tracking for Efficiency: Postponing decisions during ambiguous periods significantly reduces computational load without compromising the final identification accuracy.
Key Results
- High Accuracy: The method achieves high identification accuracy, exceeding 90% in simulations and experiments, even with short interambiguity times (as low as 3.5 seconds).
- Robustness to Noise and Low Resolution: The system demonstrates good performance even with noisy sensor data and the lower resolution of privacy-preserving motion cameras (achieving over 80% accuracy).
- Real-time Performance: The worst-case running time was approximately 6x faster than real-time, indicating its suitability for practical applications.
- Simulation vs. Experiment Correlation: Experimental results closely followed the trends observed in simulations, validating the simulation models.
- Ambiguity Resolution: The system effectively disambiguates between people after proximity or static ambiguities, with an average accuracy of 84.37% in challenging 2-person ambiguity tests.
Limitations and Future Work
- Dependence on Vision Subsystem: The localization accuracy is fundamentally limited by the performance of the underlying camera network's person-detection and tracking modules. The paper suggests that improving these modules would directly enhance the overall system.
- Static Ambiguity Handling: While the lazy tracking approach helps, the system's ability to recover from static ambiguities (where a person is motionless) is still a challenge, though the inertial sensors are expected to aid in disambiguation.
- Sensor Synchronization: While the system is designed to be robust to asynchronous sampling and synchronization disparities, significant clock drift could still impact performance.
- Future Work: The authors suggest that the probabilistic formulation is general and can be extended to other sensing modalities. Further improvements could involve integrating more sophisticated multi-target tracking techniques to generate the initial track hypotheses.
Summary updated