Perceiving Humans: from Monocular 3D Localization to Social Distancing

Lorenzo Bertoni, Sven Kreiss, Alexandre Alahi

Abstract

Perceiving humans in the context of Intelligent Transportation Systems (ITS) often relies on multiple cameras or expensive LiDAR sensors. In this work, we present a new cost-effective vision-based method that perceives humans' locations in 3D and their body orientation from a single image. We address the challenges related to the ill-posed monocular 3D tasks by proposing a neural network architecture that predicts confidence intervals in contrast to point estimates. Our proposed solution (i) is privacy-safe, (ii) works with any fixed or moving cameras, and (iii) does not rely on ground plane estimation. We demonstrate the performance of our method with respect to three applications: locating humans in 3D, detecting social interactions, and verifying the compliance of recent safety measures due to the COVID-19 outbreak.

Introduction

Localizing humans in 3D space is a cornerstone of intelligent transportation and robotics. Traditionally, this requires expensive LiDAR sensors or complex multi-camera (stereo) setups. Doing this with a single camera—monocular 3D localization—is inherently "ill-posed": without depth information, a system cannot easily distinguish between a tall person far away and a short person standing close.

Furthermore, the COVID-19 pandemic highlighted a flaw in how we measure "social distancing." Most automated systems use a simple distance radius, but the risk of viral transmission is significantly higher when people are facing each other and talking. To truly monitor safety, a system needs to perceive not just where people are, but how they are oriented and whether they are interacting.

Methods: The MonoLoco++ Framework

The authors propose a three-stage pipeline called MonoLoco++ designed to navigate the ambiguities of monocular vision:

  1. 2D Pose Detection: The system uses an off-the-shelf detector (PifPaf) to extract 2D body keypoints. This makes the system "privacy-safe," as subsequent processing only requires anonymous coordinates.
  2. 3D Estimation: These 2D poses are fed into a lightweight feed-forward neural network. The model uses spherical coordinates (radial distance, azimuthal angle, and polar angle) to disentangle depth ambiguity from image projection.
  3. Uncertainty Modeling: The network predicts a Laplace-based confidence interval. It uses Monte Carlo (MC) Dropout to account for both "epistemic" uncertainty (model ignorance) and "aleatoric" uncertainty (noise in the data).
  4. Social Interaction Detection: By combining 3D locations and orientations, the system identifies F-formations—spatial patterns humans naturally form when interacting.
The MonoLoco++ architecture: a three-stage pipeline from 2D pose detection to 3D localization and F-formation estimation.

Key Results and Performance

MonoLoco++ was evaluated on several benchmarks, including the KITTI dataset and nuScenes. A key insight from the research is the "Task Error" Bound: the authors mathematically modeled the intrinsic error caused by human height variation. They found that even a "perfect" monocular system will have an error of about 1 meter at a 20-meter distance simply because humans aren't all the same height.

Average localization error (ALE) vs. ground-truth distance. MonoLoco++ tracks the theoretical 'Task error' bound and outperforms other monocular baselines.

As shown in the performance analysis, MonoLoco++ outperformed existing monocular methods (like MonoPSR and MonoDIS) and achieved results comparable to some stereo-camera methods. Other highlights include:

  • Social Interaction: The system achieved 91.5% accuracy in recognizing "talking" activities on the Collective Activity Dataset.
  • Efficiency: The 3D estimation part of the pipeline is extremely fast, taking only about 10ms per frame on a standard GPU.
  • Privacy: Because the core logic operates on 2D keypoints rather than pixels, raw images never need to leave the camera sensor.

Limitations and Future Work

While robust, the system's performance is currently capped by the quality of the 2D pose detector; heavy occlusion can lead to missing keypoints and failed 3D estimation. Additionally, the current model processes frames independently. Future work could involve incorporating temporal information (video tracking) to smooth out 3D estimates and learning social space parameters dynamically based on the environment, such as distinguishing between a crowded subway and a park.

Summary updated

Abstract

Perceiving humans in the context of Intelligent Transportation Systems (ITS) often relies on multiple cameras or expensive LiDAR sensors. In this work, we present a new cost-effective vision-based method that perceives humans' locations in 3D and their body orientation from a single image. We address the challenges related to the ill-posed monocular 3D tasks by proposing a neural network architecture that predicts confidence intervals in contrast to point estimates. Our neural network estimates human 3D body locations and their orientation with a measure of uncertainty. Our proposed solution (i) is privacy-safe, (ii) works with any fixed or moving cameras, and (iii) does not rely on ground plane estimation. We demonstrate the performance of our method with respect to three applications: locating humans in 3D, detecting social interactions, and verifying the compliance of recent safety measures due to the COVID-19 outbreak. We show that it is possible to rethink the concept of "social distancing" as a form of social interaction in contrast to a simple location-based rule. We publicly share the source code towards an open science mission.
BibTeX
@article{bertoni2022perceiving,
  author = {Lorenzo Bertoni and Sven Kreiss and Alexandre Alahi},
  title = {{Perceiving Humans: from Monocular 3D Localization to Social Distancing}},
  journal = {IEEE Trans. Intell. Transp. Syst.},
  year = {2022},
  url = {https://openreview.net/forum?id=lV_n80iFg5},
}
Figure