TL;DR
This paper tackles the problem of "smeared points," an artifact in consumer-grade depth sensors that creates false surfaces and degrades 3D data. The authors introduce a novel self-annotation method that leverages multi-view geometric consistency to automatically label these problematic points, enabling the training of a robust removal network without manual annotation. This approach significantly improves the quality of depth data for downstream 3D applications.
What Problem Does This Paper Address?
Consumer-grade depth sensors, like those found in many smartphones and robotics platforms, have become increasingly accurate. However, they still suffer from a common artifact known as "smeared points." These are invalid depth measurements that appear as interpolations between foreground and background objects, effectively creating fictitious surfaces. This is particularly problematic at object boundaries where depth discontinuities exist. Traditional outlier removal methods often fail because they can mistakenly remove valid surface points. While deep learning offers a promising solution, obtaining sufficient annotated data for training smeared point removal models is a significant challenge due to the expense and time involved in manual annotation. This paper aims to bridge this gap by proposing a method that can automatically generate training data for smeared point removal.
Methods
The core of this paper is a self-annotation framework designed to automatically identify and label smeared points. This framework operates in two main stages:
- Geometric Label Generation: This stage leverages multi-view consistency. The key idea is that true 3D points on an object will remain consistent across different viewpoints, even as the sensor moves. In contrast, smeared points, being interpolations, will shift their apparent location or disappear as the viewpoint changes.
- Multi-View Evidence: By observing the same scene from multiple viewpoints, the system can identify pixels that are consistently observed. These are labeled as "valid" points.
- See-Through Evidence: This technique exploits the fact that smeared points often lie along rays that pass through foreground objects to the background. If a ray from one viewpoint appears to pass through a detected 3D point and then hits another point, or if there's an absence of a point along that ray where one is expected, it's strong evidence that the initial point is smeared. This is further divided into "See-Through Behind" (positive evidence) and "See-Through Empty" (negative evidence, using a conical section to avoid false positives).
- Algorithm 1 formalizes this process, using geometric evidence from multiple frames to label pixels as "valid," "smeared," or "unknown."
- Pixel Classifier Training: The automatically generated geometric labels are then used to train a supervised single-frame smeared point removal classifier. The authors opt for a 2D segmentation network (specifically, a U-Net architecture with a ResNet-34 backbone) for this task. This choice is motivated by its lightweight nature, the direct availability of depth maps, and the fact that smeared points often deviate along the viewing ray (z-axis). The classifier takes a depth map and associated viewing ray information (calculated using surface normals) as input and outputs a probability map indicating smeared points. A binary cross-entropy loss function is employed, with weights and a confidence score derived from the geometric labels to improve robustness.
Datasets
- AzureKinect Dataset (New Benchmark): The authors introduce a new dataset collected using an Azure Kinect sensor. It comprises 50 indoor and outdoor scenes, captured with hand-held movement, resulting in 1936 pairs of color and depth frames (1920x1080 resolution). Crucially, this dataset includes pose information for consecutive frames, which is essential for the multi-view self-annotation process. While the dataset is large in terms of raw frames, only 11 typical frames were manually annotated to serve as ground truth for evaluation, acknowledging the difficulty of manual annotation.
The paper also references and compares against existing datasets like LaMAR, NYU Depth V2, and ScanNet, noting that even these "well-reconstructed" datasets contain smeared points.
Key Insights and Ideas
- Self-Annotation is Viable: The most significant insight is that geometric consistency across multiple viewpoints can reliably generate training labels for smeared points, circumventing the need for expensive manual annotation. This opens up possibilities for training on much larger datasets.
- Smeared Points are Distinct: The paper highlights that smeared points have different statistical properties (e.g., one-sided distributions at object boundaries) compared to other types of depth noise, making them a unique challenge.
- Multi-View Consistency as Ground Truth: The core idea that consistent 3D points across views are valid, while inconsistent or interpolated points are likely smeared, is a powerful and intuitive principle.
- "See-Through" as a Powerful Cue: The "See-Through Behind" and "See-Through Empty" heuristics provide strong geometric evidence for identifying smeared points, especially in scenarios with significant depth discontinuities.
- 2D Networks Suffice: The finding that a 2D segmentation network, augmented with geometric cues like viewing ray information, is sufficient for this task is important for efficiency.
Key Results
- High mAP Performance: The self-annotated UNet model achieved the highest Mean Average Precision (mAP) on their AzureKinect dataset compared to traditional filters (Median Filter, Statistical Filter) and other methods like DeepDD, DeepToF, DeepLabV3+, and Segformer. The UNet model achieved an mAP of 0.775 (with uniform weighting) and 0.771 (with confidence score weighting).
- Improved Alignment: Qualitative results demonstrate that using their trained network as a preprocessor for 3D reconstruction significantly improves the alignment and fidelity of point clouds compared to no filtering or standard outlier filters.
- Confidence Score Benefits: Using a confidence score derived from multi-view evidence (Eq. 1) for labeling resulted in a 4% higher mAP compared to uniform weighting.
- Input Modality Matters: Incorporating the viewing ray inner product map (
w) alongside depth significantly boosted performance, indicating its importance for the classifier.
Limitations and Future Work
- Static Scenes for Training: The current self-annotation pipeline requires static scenes for training, limiting the diversity of training data.
- Partial Annotations: The self-annotated labels are partial, meaning not all smeared points are perfectly identified. Further investigation into mechanisms that encourage models to connect and attribute predictions among similar 3D geometry structures is needed.
- Pose Estimation Errors: Incorrect pose estimation can lead to errors in the self-annotation process and, consequently, in the trained model. The authors explored retraining with pseudo-labels generated from valid points only, which showed a modest improvement.
- Future Work: The authors suggest exploring more robust pose estimation and investigating methods to better leverage the partial self-annotated labels. They also highlight the potential for this method to be used as a preprocessor for real-time 3D reconstruction.
Summary updated