When a robot or a computer vision system looks at an object, it never sees the whole thing. It sees a "partial shell"—the surfaces facing the camera. However, the digital references we use to identify these objects, such as CAD models or 3D meshes, are complete volumes.
This discrepancy creates a fundamental challenge in 3D registration: if you try to align a complete 3D model to a partial sensor observation, the "hidden" parts of the model (the back and interior) act as geometric noise. They create false correspondences that pull the alignment away from the truth. A growing body of research suggests that the key to robust 3D perception is not just better matching, but smarter pruning: filtering the model to see only what the sensor sees.
The Partial View Problem
Traditional registration algorithms, like Iterative Closest Point (ICP), attempt to minimize the distance between a source model and a target observation. But as Mohammad Altillawi and colleagues note in CE-NPBG, standard projections often suffer from "visibility mismatch." In autonomous driving, for instance, a LiDAR scan might contain millions of points, many of which are occluded by buildings or vehicles from the perspective of a specific camera.
If the system tries to match a 2D image against the entire 3D point cloud, occluded geometry incorrectly appears in the foreground. This isn't just a rendering issue; it’s a mathematical one. When the back of a chair is included in a calculation to align the front of a chair, the optimization becomes biased.
Filtering for Coherence
To solve this, researchers have moved toward "viewpoint-aware" pipelines. The intuition is simple: before you calculate how well a model fits a scene, you must first ask, "Given my current best guess of the camera's position, which parts of this model would actually be visible?"
Ruizhe Wang 0002 and Jongmoo Choi pioneered this with "contour coherence" in 3D Modeling from Wide Baseline Range Scans Using Contour Coherence. By matching observed silhouettes to predicted 3D contours and pruning occluded points, they enabled accurate modeling from as few as four scans. This pruning ensures that only points physically capable of being observed in both viewpoints are used for alignment, preventing the "false correspondences" that plague wide-baseline registration.
Hidden Point Removal in Practice
This concept of visibility filtering has become a cornerstone for tracking complex or unseen objects. In the realm of soft robotics, Shiyang Lu and Rebecca Kramer-Bottiglio applied "hidden point removal" to track tensegrity robots. Because these robots consist of many overlapping rods and cables, self-occlusion is constant. By filtering the reference mesh to match the camera's perspective, their system maintains sub-centimeter accuracy even when parts of the robot "ghost" behind others.
The ZeroPose framework, developed by Jianqiu Chen and Zhenyu He 0001, takes this a step further for zero-shot object detection. Instead of retraining a model for every new object, ZeroPose uses a CAD model as a prompt. It predicts the camera's viewpoint first, then uses that prediction to strip away the back-facing and occluded points of the CAD model. This "Discovery-Orientation-Registration" pipeline ensures the final registration step only compares the visible "shell" of the CAD model to the partial point cloud from the sensor, leading to a 50x speedup over methods that don't account for visibility.
Beyond Rigid Objects
The utility of viewpoint-aware filtering extends to dynamic, large-scale environments. In HiSC4D, Yudi Dai and Cheng Wang use a viewpoint-aware loss function to capture human-environment interactions. By filtering out non-visible mesh vertices before performing point-cloud matching, they can accurately register human motion within sparse LiDAR data.
Whether it is a tensegrity robot, a new factory part, or a pedestrian on a street, the underlying principle remains the same: 3D registration is most accurate when the digital model is forced to share the sensor's limited perspective. By acknowledging what cannot be seen, these systems achieve a much clearer understanding of what is there.