Depth completion—the task of turning sparse, "holey" sensor data (like LiDAR) into a dense, high-resolution depth map using an RGB image as a guide—is a cornerstone of autonomous navigation and AR. Traditionally, researchers have treated this as either a regression problem (predicting a continuous value) or a classification problem (assigning pixels to discrete depth "bins").
While classification-based methods have shown great promise, they suffer from a fundamental flaw: they rely on pre-defined, discrete depth categories shared across all pixels. This leads to "depth smearing," where the boundaries between a foreground object and the background become a blurry, stepped mess. A new paper, "Learning Pixel-wise Continuous Depth Representation via Clustering for Depth Completion," introduces a framework called CluDe to solve exactly this.
TL;DR
CluDe (Clustering-based Depth completion) replaces fixed depth "bins" with learnable, scene-adaptive depth centers. By combining a Clustering Transformer with a hierarchical offset estimation module, it generates a pixel-wise continuous depth representation that significantly reduces smearing at object boundaries and achieves state-of-the-art performance on the VOID dataset.
What Problem Does This Paper Address?
Most classification-based depth completion models use a fixed set of depth "categories" (e.g., 80 planes at specific distances). This approach assumes the depth distribution is uniform and that every pixel can be represented by the same set of discrete values.
In reality, depth is continuous. When a model tries to force a pixel into a discrete bin that doesn't quite fit, it creates artifacts. This is most visible at object edges—think of a person standing in front of a wall. Fixed-bin models often create "streaks" or "stripes" between the person and the wall because they lack the resolution to represent the sharp transition. CluDe aims to bridge the gap between the stability of classification and the precision of continuous regression.
Methods
The authors propose a two-stage framework:
- Clustering Transformer: Instead of using pre-defined depth bins, CluDe uses a transformer to learn a set of "depth centers" directly from the data. These centers are feature vectors that represent the most likely depth values for a specific input scene. This makes the "bins" dynamic and adaptive rather than static.
- Hierarchical Translation Module: This is the core innovation. Since the learned centers are still technically discrete, CluDe estimates pixel-wise offsets. These offsets "nudge" the depth centers along the depth axis for every single pixel. This transforms a discrete representation into a continuous one, allowing the model to represent any depth value within a range.
- Prune Translation Block: LiDAR data is notoriously noisy. CluDe includes a filtering mechanism that uses the initial estimated depth as a reference to identify and "prune" outliers in the sparse input map, ensuring the final refinement isn't led astray by sensor errors.
Datasets
The model was evaluated on two major benchmarks:
- KITTI Depth Completion: The industry standard for outdoor autonomous driving scenes.
- VOID (Visual Odometry Indoor Dataset): A challenging dataset containing both indoor and outdoor scenes with varying levels of sparsity (from 1500 points down to just 150 points per frame).
Key Insights and Ideas
- The "Aha" Moment for Offsets: The authors realized that you don't need an infinite number of bins to get continuous depth. You just need a few smart "centers" and the ability to shift them locally. This "classification + offset" approach provides the convergence speed of classification with the edge-fidelity of regression.
- Scene-Adaptive Bins: By using a transformer to cluster tokens into depth centers, the model effectively "re-discretizes" the world for every image. If a scene is mostly close-up, the centers cluster near the camera; if it's an open highway, they spread out.
- Global vs. Local Offsets: The model calculates both a global offset (to correct the entire scene's depth continuity) and local offsets (to fine-tune details around complex object boundaries).
Key Results
- State-of-the-Art (SOTA): CluDe achieved the top spot on the VOID dataset, outperforming previous leaders like CostDCNet.
- Boundary Excellence: Qualitatively, CluDe produces much cleaner edges. In head-to-head comparisons with standard classification models, CluDe effectively eliminated the "stripe" artifacts between foreground and background objects.
- Efficiency: On the KITTI test set, CluDe achieved a Mean Absolute Error (MAE) of 197.91 mm, outperforming other classification-based methods while maintaining a competitive latency of 0.14s per image—faster than many high-end regression models like NLSPN (0.22s).
Limitations and Future Work
Despite its success, CluDe struggles with the "long-tail" problem. In datasets like KITTI, very few pixels represent distances beyond 65 meters. Because there is so little data in this "tail," the clustering mechanism has a harder time learning accurate representations for extreme distances.
The authors suggest that future work should focus on improving performance in these tail categories and further optimizing the transformer architecture to reduce the computational footprint for real-time edge deployment.
Summary updated