When a computer vision model predicts the shape of a room, the pose of a pedestrian, or the 3D occupancy of a driving scene, it must reason about space that it cannot directly see. Objects block other objects, surfaces turn away from the camera, and boundaries cut off the field of view.
Evaluating these models seems straightforward: compare the model's predictions against a ground-truth 3D scan or a human-annotated mask. However, standard evaluation metrics suffer from a fundamental flaw. By treating all pixels or voxels equally, traditional metrics introduce severe biases that either reward models for cheating or penalize them for being correct.
To build models that truly understand the physical world, researchers are shifting toward visibility-aware evaluation. This paradigm explicitly decouples what a model understands about visible surfaces from what it infers about hidden or unobserved spaces.
The Illusion of High Accuracy
The primary failure mode of standard metrics—such as Average Precision (AP) or mean Intersection over Union (mIoU)—is that visible regions typically dominate the pixel or voxel count. If a vehicle is 90% visible and 10% occluded by a lamppost, a model can completely ignore the occluded section and still achieve a 90% accuracy score.
As pointed out by Ziyu Jiang et al. in their work on indoor scene understanding, Peek-a-Boo: Occlusion Reasoning in Indoor Scenes With Plane Representations, this dominance masks critical failures in geometric reasoning. A model can score exceptionally well overall while failing entirely to predict the hidden geometry behind foreground objects.
Similarly, in video amodal segmentation, Finlay G C Hudson et al. demonstrated in Real-VAS: a Realworld Video Amodal Segmentation dataset that standard mIoU yields misleadingly high scores. Because the visible parts of an object skew the metric, the score fails to reflect whether a model actually understands object permanence and containment when things disappear from view.
When supervision and evaluation are derived purely from single-view images, as explored by Haochen Yu et al. in ProtoCar: Learning 3D Vehicle Prototypes from Single-View and Unconstrained Driving Scene Images, the metrics become inherently biased toward visible surfaces. The accuracy of the reconstructed geometry in occluded or entirely unseen regions remains unverified, leaving a massive blind spot in performance assessment.
The Sensor Line-of-Sight Problem
While standard metrics allow models to hide their failures in occluded zones, they also cause the opposite problem: unfairly penalizing models for predicting valid geometry that the ground-truth sensors missed.
Consider a 3D laser scanner capturing a scene. The scanner can only record surfaces in its direct line of sight; it cannot see through a car or a wall. If an advanced model correctly infers that a complete 3D object exists behind an obstacle, a naive evaluation metric will compare this prediction against an empty or missing region in the ground-truth scan. The metric will then penalize the model for a "false positive."
Sensor Ray: ========> [ Visible Surface ] x x x x x (Occluded/Hole in GT)
Model Prediction: [ Visible Surface ] [ Inferred Hidden Geometry ]
Standard Metric: (Match: OK) (No GT: PENALIZED)
Visibility Masking: (Match: OK) (Ignored or Evaluated Separately)
To prevent this bias, early multi-view stereopsis benchmarks implemented visibility masks. As shown by Rasmus Ramsbøl Jensen et al. in Large Scale Multi-view Stereopsis Evaluation, applying visibility masks to the evaluation space ensures that reconstruction algorithms are only penalized for missing geometry in regions that were successfully captured by the reference sensor. This insight is echoed by Mohamed Sayed et al. in DoubleTake: Geometry Guided Depth Estimation, where tighter visibility masks prevent models from being judged on their ability to replicate accidental "holes" in a ground-truth scan rather than their actual reconstruction quality.
In autonomous driving, this issue is magnified. In Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving, Xiaoyu Tian et al. emphasize that treating "unobserved" as a distinct state is vital. By generating visibility-aware labels, their benchmark ensures that models are not penalized for failing to predict occupancy in voxels that are physically occluded from the sensor's perspective, leading to a much fairer and more accurate assessment of a vehicle's perception system.
How Visibility-Aware Metrics Work
To resolve these biases, researchers have designed evaluation frameworks that explicitly isolate, stratify, or penalize models based on visibility states.
1. Targeted Occlusion Sub-Metrics
Instead of averaging across the entire image or volume, visibility-aware evaluation isolates the hidden regions. For example, Finlay G C Hudson et al. proposed stricter metrics like $mIoU_{occ}$ (which evaluates performance exclusively on occluded pixels) and $mIoU_{ffo}$ (which evaluates fully occluded frames). By stripping away the visible pixels, these metrics force the score to reflect only the model's amodal reasoning capabilities.
2. Visibility Stratification (Decile Averaging)
Traditional metrics favor models that perform exceptionally well on highly visible objects, masking poor performance under heavy occlusion. To counter this, the authors of WALDO: Where Unseen Model-based 6D Pose Estimation Meets Occlusion proposed an occlusion-aware metric that groups test instances into visibility deciles (e.g., 0-10% visible, 10-20% visible, etc.) and averages recall across them. This prevents easy, highly visible cases from inflating the final score, providing a balanced measure of real-world robustness. However, as noted by Xiaolin Song et al. in Progressive Refinement Network for Occluded Pedestrian Detection, these thresholds must be chosen realistically; including instances with near-zero visibility can introduce uninformative noise that even humans cannot reliably annotate.
3. Accounting for Out-of-Frame and Absent Features
Visibility awareness also applies to what should not be predicted. In human pose estimation, standard metrics often ignore keypoints that fall outside the image frame. In ProbPose: A Probabilistic Approach to 2D Human Pose Estimation, Miroslav Purkrábek et al. highlight that failing to penalize models for predicting locations for absent or out-of-frame joints creates a major performance gap. A visibility-aware framework holds models accountable for explicitly distinguishing between visible, occluded, and entirely absent features.
4. Cross-Sensor Degradation Metrics
In collaborative perception (e.g., infrastructure-to-vehicle communication), visibility changes dynamically based on which sensors are active. Xiaofei Zhang et al. introduced a metric, $\xi_p$, in InScope: A new real-world 3D infrastructure-side collaborative perception dataset for open traffic scenarios. This metric quantifies anti-occlusion capability by measuring the exact performance degradation when a system relies solely on a single occluded sensor versus fused, multi-perspective data, standardizing how we benchmark robustness.
Why This Shift Matters
Visibility-aware evaluation changes how models behave during training. When a metric penalizes a model for failing in hidden zones or for hallucinating geometry where sensors were blind, the loss functions derived from these metrics force neural networks to learn true spatial and geometric priors. Instead of memorizing surface textures or overfitting to the line-of-sight artifacts of a specific sensor setup, models are incentivized to understand object permanence, structural continuity, and the fundamental physics of 3D space.
Key Foundational Work
Peek-a-Boo: Occlusion Reasoning in Indoor Scenes With Plane Representations (2020) — contributed 2 concept elements to this direction.
Large Scale Multi-view Stereopsis Evaluation (2014) — contributed 1 concept elements to this direction.
Progressive Refinement Network for Occluded Pedestrian Detection (2020) — contributed 1 concept elements to this direction.