Visual-GPS: Ego-Downward and Ambient Video Based Person Location Association

Liang Yang, Hao Jiang, Zhouyuan Huo, Jizhong XiaoWorkshops @ CVPR 2019 · OpenReview

Abstract

In a crowded and cluttered environment, identifying a particular person is a challenging problem. Current identification approaches are not able to handle the dynamic environment. In this paper, we tackle the problem of identifying and tracking a person of interest in the crowded environment using egocentric and third person view videos. We propose a novel method (Visual-GPS) to identify, track, and localize the person, who is capturing the egocentric video, using joint analysis of imagery from both videos. The output of our method is the bounding box of the target person detected in each frame of the third person view and the 3D metric trajectory. At glance, the views of the two cameras are quite different. This paper illustrates an insight into how they are correlated. Our proposed method uses several difference clues. In addition to using RGB images, we take advantage of both the body motion and action features to correlate the two views. We can track and localize the person by finding the most "correlated" individual in the third view. Furthermore, the target person's 3D trajectory is recovered based on the mapping of the 2d-3D body joints. Our experiment confirms the effectiveness of ETVIT network and shows 18.32 % improvement in detection accuracy against the baseline methods.

Introduction: The Challenge of Cross-View Association

Identifying a specific individual in a sea of faces within a crowded room is a significant challenge for surveillance systems. While fixed ambient cameras (third-person view) provide a global perspective, they lack the inherent context to know which individual corresponds to a specific user or wearable device. Conversely, wearable cameras (egocentric view) capture the user's direct experience but lack global spatial context.

Overview of the Visual-GPS system linking egocentric and third-person views through motion and action correlation.

Existing methods for linking these views often rely on appearance-based matching, such as clothing color or texture. However, these methods fail in "uniform" crowds—such as corporate offices or marathons—where many people dress similarly, or in changing lighting conditions. Visual-GPS addresses this by focusing on body motion and 3D pose, providing a robust biometric signature that persists even when appearance cues are ambiguous.

Methodology: The ETVIT Architecture

The core of the Visual-GPS system is the ETVIT (Ego-downward and Third-view Identification and Tracking) network. This deep neural network architecture is designed to bridge the gap between the disparate perspectives of wearable and ambient cameras.

Key technical components include:

  • Ego-Downward Perspective: Instead of a standard forward-looking camera, the wearer uses a head-mounted camera angled downward. This captures the wearer's own limbs, providing a direct visual link to the skeletal motion seen by the surveillance camera.
  • Four-Channel DNN: The ETVIT model processes four distinct data streams to find correlations:
  • Third-view Action: Uses a 3D ResNet-18 to classify body actions.
  • Third-view Motion: Uses a 2D ResNet-50 on optical flow to track 2D translation.
  • Ego-view Action: Analyzes the wearer's limb movements.
  • Ego-view Motion: Estimates the wearer's own movement (odometry).
  • 3D Pose Integration: By utilizing the SMPL (Skinned Multi-Person Linear) model, the system estimates 3D body joints. This allows local egocentric movements to be translated into a global coordinate system shared by the surveillance camera.
The ETVIT network architecture detailing the multi-stream processing of action, motion, and pose for joint person identification.

Datasets and Scenarios

To validate the system, the researchers developed the Visual-GPS Dataset, consisting of 40 pairs of synchronized egocentric and third-person videos. This dataset includes over 100,000 image pairs featuring more than 20 different candidates across five challenging scenarios. Notably, it includes "same-dressing" scenarios where multiple individuals wear identical clothing, specifically designed to test the limits of motion-based identification over appearance-based methods.

Key Insights: Motion as a Biometric

The fundamental insight of this research is that motion serves as a unique biometric signature. While two people might wear identical red shirts, their gait, posture, and specific actions—such as checking a watch or turning—are distinct. By looking downward, the ego-camera captures these unique limb movements, which can then be mapped to the skeletal motion observed in the third-person view. This approach effectively bridges the coordinate gap, as 3D pose estimation helps align the wearer's orientation with the room's global field of view.

Results and Performance

The ETVIT model demonstrated significant performance gains over traditional methods:

  • Verification Accuracy: The system achieved an accuracy of 74.22%, representing an 18.32% improvement over the best baseline (a temporal Siamese flow model).
  • Component Analysis: The "Action" component (identifying specific movements) was found to be slightly more effective (72.5%) than the "Translation" component (70.03%), though the combination of both yielded the highest accuracy.
  • 3D Trajectory Recovery: The system successfully reconstructed 3D metric trajectories of the target person, proving more stable than standard Visual Odometry (VO), which often suffers from significant drift over time.

Conclusion and Future Work

Visual-GPS provides a robust framework for person identification and tracking in cluttered environments by prioritizing motion and pose over appearance. While the system currently requires a specific ego-downward camera mount, future research may explore extracting these motion features from more common wearable positions, such as chest mounts or smart glasses. Additionally, integrating more sophisticated multi-person tracking filters could further improve performance in extremely dense crowds where occlusions are frequent.

Summary updated

Abstract

In a crowded and cluttered environment, identifying a particular person is a challenging problem. Current identification approaches are not able to handle the dynamic environment. In this paper, we tackle the problem of identifying and tracking a person of interest in the crowded environment using egocentric and third person view videos. We propose a novel method (Visual-GPS) to identify, track, and localize the person, who is capturing the egocentric video, using joint analysis of imagery from both videos. The output of our method is the bounding box of the target person detected in each frame of the third person view and the 3D metric trajectory. At glance, the views of the two cameras are quite different. This paper illustrates an insight into how they are correlated. Our proposed method uses several difference clues. In addition to using RGB images, we take advantage of both the body motion and action features to correlate the two views. We can track and localize the person by finding the most "correlated" individual in the third view. Furthermore, the target person's 3D trajectory is recovered based on the mapping of the 2d-3D body joints. Our experiment confirms the effectiveness of ETVIT network and shows 18.32 % improvement in detection accuracy against the baseline methods.
BibTeX
@inproceedings{yang2019visual,
  author = {Liang Yang and Hao Jiang and Zhouyuan Huo and Jizhong Xiao},
  title = {{Visual-GPS: Ego-Downward and Ambient Video Based Person Location Association}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  url = {https://openreview.net/forum?id=YcQ8obXZiX4d},
}
Figure