When reconstructing a 3D scene, tracking an object across multiple cameras, or synthesizing a novel viewpoint, a fundamental challenge is combining information from different perspectives. This process, known as feature aggregation, takes 2D image data from various viewpoints and fuses it into a unified 3D representation.
Traditional multi-view architectures often rely on uniform averaging or variance-based fusion. As noted by Zhang et al. (2020), these approaches implicitly assume universal visibility—the idea that every camera has an unobstructed line of sight to every point in the scene. In reality, complex environments are full of occlusions. When a system uniformly aggregates features across all views, cameras whose sightlines are blocked by foreground objects contribute irrelevant visual data.
This "feature pollution" introduces severe noise, leading to ghosting artifacts, blurred boundaries, and geometric distortion. As Bozic et al. (2021) point out, treating all visual observations with equal importance allows low-quality or occluded data to pollute the aggregated feature volume, reducing the overall accuracy of the 3D model.
To solve this, modern architectures incorporate visibility-aware feature aggregation. By explicitly modeling whether a spatial point is visible to a given camera, these systems can dynamically weight or gate incoming features, ensuring that only clean, unoccluded data shapes the final reconstruction.
Quantifying Visibility
To prevent feature pollution, a system must determine if a 3D point is visible from a specific camera perspective before or during the feature blending process. Researchers have developed several strategies to estimate this visibility, ranging from geometric heuristics to learned functions.
Geometric and Surface Orientation Priors
One intuitive approach leverages the geometric relationship between the camera and the target surface. In multi-view human and head capture, Chen et al. (2023) and Li et al. (2023) weight feature fusion by calculating the alignment between a 3D point's surface normal and the camera's viewing direction. If a surface faces away from a camera, or is positioned at an oblique angle, that camera's contribution is suppressed. This prevents feature bleeding across complex, non-convex geometries like ears and necks. Similarly, Li et al. (2024) restrict 3D feature aggregation strictly to views with a direct line of sight determined by a geometric proxy, ensuring that occluded viewpoints do not degrade the reconstruction of visible facial regions.
Learned Visibility and Transmittance Functions
Simple geometric heuristics often fail when foreground objects block a target point further down the line of sight, a limitation highlighted by Adam et al. (2019). To handle arbitrary occlusions, generalizable neural rendering models learn to predict visibility directly from scene context or feature consistency.
In the Neural Rays (NeuRay) framework, Liu et al. (2022) explicitly predict a visibility function for each input ray. By treating visibility as a primary model output rather than a passive byproduct of density estimation, the system ignores inconsistent features from occluded regions, significantly improving reconstruction quality. Alternatively, Flynn et al. (2019) incorporate net transmittance and accumulated alpha directly into the optimization update rules for multiplane images. This allows the model to mathematically reason about occlusions, correctly placing textures behind thin structures and complex geometry that typically cause standard image-matching approaches to fail.
How Visibility Guides Feature Fusion
Once visibility is quantified—whether as a binary gate or a continuous probability score—it fundamentally changes how features are aggregated. Instead of blindly mixing pixels, the network uses visibility as a reliability metric to execute two key behaviors: selective gating and adaptive fallback.
Selective Gating
In novel view synthesis and 3D scene understanding, visibility scores act as a filter to block corrupted data. Chen et al. (2023) utilize predicted visibility probabilities to selectively aggregate features only from views with a clear line of sight, eliminating the ghosting and "floater" artifacts that plague sparse-view neural rendering.
This gating mechanism is equally vital for semantic and language features. In open-vocabulary 3D segmentation, Li et al. (2025) employ a visibility-aware gating mechanism based on rendering weights. This ensures that 2D language features are only lifted to the specific 3D primitives that contribute to the visible surface, preventing semantic information from bleeding across depth layers into occluded background objects.
Adaptive Fallback to Structural Priors
When a region is fully visible, the model can safely rely on highly detailed, local image features. However, when the visibility score indicates an occlusion, the model must adapt. Rather than incorporating corrupted local texture data, visibility-aware networks smoothly transition to relying on global context, symmetry, or structural priors.
- Facial Landmark Tracking: Wu et al. (2015) use visibility probabilities to adaptively suppress unreliable local texture features when a landmark is obscured, instead determining its position using the global geometric configuration of other visible landmarks.
- Human Mesh Reconstruction: The authors of MeshPose (2024) implement a visibility-weighted fusion strategy that dynamically balances local 2D vertex detections with global 3D pose estimates. When a body part is hidden or truncated, the system automatically falls back to an amodal 3D regression.
- Interacting Hands: To handle the extreme self-occlusions of overlapping hands, Li et al. (2024) propose a feature fusion mechanism that adaptively weights local pixel-aligned features against symmetric structural features and global appearance summaries based on 3D visibility, preventing blurry artifacts in hidden regions.
By transforming feature aggregation from a passive averaging operation into an active, visibility-gated selection process, these methods allow neural models to maintain geometric and semantic integrity across complex, highly occluded environments.
Key Foundational Work
Robust Facial Landmark Detection Under Significant Head Poses and Occlusion (2015) — contributed 2 concept elements to this direction.