Perceiving Humans: From Monocular 3D Localization to Social Distancing

Lorenzo Bertoni, Sven Kreiss, Alexandre Alahi

TL;DR

This paper introduces MonoLoco++, a deep learning framework that estimates 3D human locations and body orientations from a single RGB image. By predicting probabilistic confidence intervals rather than simple point estimates, the system navigates the inherent ambiguities of monocular vision to accurately detect social interactions and monitor social distancing compliance.

What Problem Does This Paper Address?

Localizing humans in 3D space is a cornerstone of intelligent transportation and robotics. Traditionally, this requires expensive LiDAR sensors or complex multi-camera (stereo) setups. Doing this with a single camera—monocular 3D localization—is "ill-posed": without depth information, the computer cannot easily distinguish between a tall person far away and a short person standing close.

Furthermore, the authors argue that the COVID-19 pandemic highlighted a flaw in how we measure "social distancing." Most automated systems use a simple distance radius, but the risk of viral transmission is significantly higher when people are facing each other and talking (due to respiratory droplets). To truly monitor safety, a system needs to perceive not just where people are, but how they are oriented and whether they are interacting.

Methods

The authors propose a three-stage pipeline called MonoLoco++:

  1. 2D Pose Detection: The system uses an off-the-shelf detector (PifPaf) to extract 2D body keypoints. This makes the system "privacy-safe," as the subsequent 3D processing only requires anonymous coordinates, not raw images.
  2. 3D Estimation (MonoLoco++): These 2D poses are fed into a lightweight feed-forward neural network. Unlike previous models, this one uses spherical coordinates (radial distance, azimuthal angle, and polar angle) to disentangle depth ambiguity from image projection.
  3. Uncertainty Modeling: Crucially, the network predicts a Laplace-based confidence interval. It doesn't just say "this person is 5 meters away"; it says "this person is 5 meters away, +/- 0.5 meters." It uses Monte Carlo (MC) Dropout to account for "epistemic" uncertainty (model ignorance) and "aleatoric" uncertainty (noise in the data).
  4. Social Interaction Detection: By combining 3D locations and orientations, the system identifies F-formations—spatial patterns humans naturally form when interacting (like standing in a circle to talk).

Datasets

The model was trained and evaluated on several key benchmarks:

  • KITTI Dataset: The gold standard for 3D object detection in autonomous driving scenarios.
  • nuScenes: Used to demonstrate the model's ability to generalize across different cities and camera setups without retraining.
  • Collective Activity Dataset: Used to evaluate the detection of social activities like "walking," "waiting," and "talking."

Key Insights and Ideas

  • The "Task Error" Bound: The authors mathematically modeled the intrinsic error caused by human height variation. They found that even a "perfect" monocular system will have an error of about 1 meter at a 20-meter distance simply because humans aren't all the same height. Their model's performance closely tracks this theoretical limit.
  • Orientation Matters: The paper shifts the focus from "distance" to "interaction." By knowing body orientation, the system can distinguish between two people walking past each other and two people standing still to talk—the latter being a much higher risk for viral transmission.
  • Uncertainty as a Feature: Instead of viewing uncertainty as a failure, the authors use it to make social interaction detection more robust. If the 3D localization is highly uncertain (e.g., due to occlusion), the system requires stronger geometric evidence before flagging a social interaction.

Key Results

  • Localization Accuracy: MonoLoco++ outperformed existing monocular methods (like MonoPSR and MonoDIS) on the KITTI dataset and achieved results comparable to some stereo-camera methods.
  • Social Interaction: The system achieved 91.5% accuracy in recognizing "talking" activities on the Collective Activity Dataset, despite not being explicitly trained on that dataset's images.
  • Efficiency: The 3D estimation part of the pipeline is extremely fast, taking only about 10ms per frame on a standard GPU, making it suitable for real-time applications.
  • Privacy: Because the core 3D logic operates on 2D keypoints rather than pixels, the system can be deployed in "privacy-by-design" architectures where raw images never leave the camera sensor.

Limitations and Future Work

  • Dependency on 2D Poses: The system's performance is capped by the quality of the 2D pose detector. If a person is heavily occluded and the 2D keypoints are missing, the 3D estimation will fail.
  • Static Social Rules: The radii used to define "social spaces" (o-space, p-space) are currently based on fixed sociological averages. Future work could involve learning these parameters dynamically based on the environment (e.g., people stand closer in a crowded subway than in a park).
  • Temporal Consistency: The current model processes frames independently. Incorporating temporal information (video tracking) could smooth out 3D estimates and reduce "flicker" in uncertainty predictions.

Summary updated

Abstract

Perceiving humans in the context of Intelligent Transportation Systems (ITS) often relies on multiple cameras or expensive LiDAR sensors. In this work, we present a new cost-effective vision-based method that perceives humans’ locations in 3D and their body orientation from a single image. We address the challenges related to the ill-posed monocular 3D tasks by proposing a neural network architecture that predicts confidence intervals in contrast to point estimates. Our neural network estimates human 3D body locations and their orientation with a measure of uncertainty. Our proposed solution (i) is privacy-safe, (ii) works with any fixed or moving cameras, and (iii) does not rely on ground plane estimation. We demonstrate the performance of our method with respect to three applications: locating humans in 3D, detecting social interactions, and verifying the compliance of recent safety measures due to the COVID-19 outbreak. We show that it is possible to rethink the concept of “social distancing” as a form of social interaction in contrast to a simple location-based rule. We publicly share the source code towards an open science mission.
BibTeX
@article{bertoni2022perceiving,
  author = {Lorenzo Bertoni and Sven Kreiss and Alexandre Alahi},
  title = {{Perceiving Humans: From Monocular 3D Localization to Social Distancing}},
  journal = {IEEE Trans. Intell. Transp. Syst.},
  year = {2022},
  url = {https://openreview.net/forum?id=lV_n80iFg5},
}

Loading related content...

Figure