Decision-Level Error Attribution and Depth-Aware Safety Evaluation of Monocular RGB Human Tracking for Autonomous Robot Navigation

Abdulahi Taiwo, Chijioke Cyriacus EkechiarXiv 2026 · arXiv

TL;DR

This paper moves beyond standard geometric accuracy metrics to evaluate how monocular RGB tracking actually affects robot safety decisions. By introducing a two-factor Shapley attribution method, the authors can pinpoint whether a "bad" decision (like an unnecessary stop or a missed danger) was caused by the safety rule's geometry or by failures in the perception pipeline, such as missed detections or poor depth estimation.

What Problem Does This Paper Address?

In autonomous navigation, robots must constantly decide whether to stop or proceed based on the distance to nearby humans. While monocular RGB cameras are cheap and popular, they struggle with depth estimation. To compensate, many systems use a "lateral-only" distance rule—essentially ignoring depth and stopping if anyone is "to the side" of the robot.

Previous studies found that this approach leads to zero missed dangers but a massive number of unnecessary stops. However, it wasn't clear if these "false alarms" were due to the camera's inaccuracy or the conservative nature of the rule itself. This paper aims to untangle these factors, providing a framework to attribute decision-level errors to their specific sources.

Methods

The authors evaluate a full perception-to-decision pipeline (shown in Figure 1) consisting of:

  1. Detection: YOLOv8m-Pose to identify humans and keypoints.
  2. Localisation: Comparing three methods: a geometric baseline (M0), a calibrated ground-plane projection (M1), and a learned depth model, Depth Anything V2 (M2).
  3. Tracking: A constant-velocity Kalman filter.
  4. Decision Rules: Comparing the "lateral-only" rule against a standard planar distance rule and a proposed "uncertainty margin" rule ($d - k\sigma < \tau$) that uses the Kalman filter's variance to buy safety.

![Fig.

  1. Evaluated chain (conceptual diagram).](/figures/art_54e891de72284ae4b86a1550c2136637)

The core innovation is the use of two-factor Shapley attribution. This mathematical framework splits the total decision error into "rule shares" and "perception shares." This allows researchers to see, for example, exactly how much of a missed danger was due to a person being missed by the detector versus being localized incorrectly.

Datasets

The study uses real-world pedestrian video from the ETH and UCY datasets. These datasets provide metric ground-plane annotations for hundreds of pedestrians. Because these are static videos, the authors used simulated stationary robots placed on a 1m grid to evaluate potential interactions. They also employed a strict calibration/evaluation split to ensure that no individual pedestrian seen during the "tuning" phase was used for the final results.

Key Insights and Ideas

  • The Lateral Rule is a Geometric Bound: The authors prove (Proposition 1) that the lateral-only distance is a mathematical lower bound on the true planar distance. This means its 0% miss rate isn't a "win" for the sensor; it's an inherent property of the geometry that comes at the cost of extreme inefficiency.
  • Detection is the Weakest Link: In these elevated views, missed detections (low recall) were the primary source of missed dangers. If the detector doesn't see a person, the safety rule can't react, regardless of how good the depth estimation is.
  • Uncertainty Margins Beat "Hacks": Instead of discarding depth information (the lateral rule), using a 2-sigma uncertainty margin on a planar distance rule provided a better trade-off. It matched the safety of the lateral rule while significantly reducing unnecessary stops.
  • Localisation Hierarchy: A simple, calibrated ground-plane mapping (M1) outperformed both the anthropometric geometric baseline and the sophisticated "Depth Anything V2" model in these specific scenes.

Key Results

  • Localisation Accuracy: The ground-plane method (M1) achieved a planar RMSE of 0.51m–0.59m, significantly better than the geometric baseline (1.39m–1.72m) and Depth Anything V2 (0.68m–0.81m).
  • Decision Efficiency: On annotated positions, the lateral rule had a stop precision as low as 5.6%, meaning nearly 95% of its stops were unnecessary.
  • Safety Trade-offs: As shown in Figure 3, the lateral rule's precision drops as the scene gets deeper, a trend that closely matches a theoretical "uniform-occupancy" model.

![Fig.

  1. Lateral rule on annotated positions (no perception).](/figures/art_451ccf17521c41ad9c01417c1855dc8c)

Limitations and Future Work

The authors acknowledge several limitations:

  • Simulated Interaction: The robots were stationary and simulated; real-world dynamic avoidance was not tested.
  • Camera Perspective: The ETH/UCY datasets use elevated, near-nadir views. Results might differ significantly for a robot-mounted camera at eye level.
  • Ground Truth Quality: The "ground truth" consists of manual annotations projected through homographies, which contains its own inherent errors.

The next steps involve repeating this protocol using motion-capture ground truth and testing from actual robot-level viewpoints to see how perspective affects the perception-to-decision error split.

Summary updated

Abstract

Monocular RGB tracking is useful for robots operating near people, but perception errors affect binary stop/proceed decisions in ways that geometric accuracy metrics do not capture. We examine how much decision error arises from the safety rule versus perception. We prove that lateral-only distance is a lower bound on planar distance: with accurate positions, it cannot miss dangers identified by the planar rule, but it produces unnecessary stops. We introduce a two-factor Shapley attribution separating rule and perception contributions, and further distinguish missed detections from localization errors. Experiments use ETH and UCY pedestrian videos with simulated stationary robots, disjoint calibration and evaluation pedestrians, and time-block bootstrap intervals. Across 898 pedestrians in three scenes, the lateral rule missed no dangers on annotated positions but achieved only 5.6-31.9% stop precision. In two scenes with RGB perception, calibrated ground-plane localization achieved 0.59 m and 0.51 m planar RMSE, compared with 1.72 m and 1.39 m for a geometric baseline and 0.81 m and 0.68 m for calibrated Depth Anything V2. Detection recall of 0.43 and 0.26 made missed detections the leading source of missed danger. A two-standard-deviation safety margin with ground-plane localization matched or improved on the lateral rule's missed-danger rate in all eight scene-threshold settings and reduced unnecessary stops in seven. All robot interactions are simulated, and outcomes are offline decision classifications, not demonstrations of collision avoidance.
BibTeX
@misc{taiwo2026decision,
  author = {Abdulahi Taiwo and Chijioke Cyriacus Ekechi},
  title = {{Decision-Level Error Attribution and Depth-Aware Safety Evaluation of Monocular RGB Human Tracking for Autonomous Robot Navigation}},
  year = {2026},
  eprint = {2610.40199},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.40199},
}
Figure