Learning Pixel-wise Continuous Depth Representation via Clustering for Depth Completion

Shenglun Chen, Hong Zhang, Xinzhu Ma, Zhihui Wang, Haojie Li

Abstract

Depth completion is a long-standing challenge in computer vision, where classification-based methods have made tremendous progress in recent years. However, most existing classification-based methods rely on pre-defined pixel-shared and discrete depth values as depth categories. This representation fails to capture the continuous depth values that conform to the real depth distribution, leading to depth smearing in boundary regions. To address this issue, we revisit depth completion from the clustering perspective and propose a novel clustering-based framework called CluDe which focuses on learning the pixel-wise and continuous depth representation. The key idea of CluDe is to iteratively update the pixel-shared and discrete depth representation to its corresponding pixel-wise and continuous counterpart, driven by the real depth distribution. Specifically, CluDe first utilizes depth value clustering to learn a set of depth centers as the depth representation. While these depth centers are pixel-shared and discrete, they are more in line with the real depth distribution compared to pre-defined depth categories. Then, CluDe estimates offsets for these depth centers, enabling their dynamic adjustment along the depth axis of the depth distribution to generate the pixel-wise and continuous depth representation. Extensive experiments demonstrate that CluDe successfully reduces depth smearing around object boundaries by utilizing pixel-wise and continuous depth representation. Furthermore, CluDe achieves state-of-the-art performance on the VOID datasets and outperforms classification-based methods on the KITTI dataset.

Introduction: The Challenge of Discrete Depth Bins

Depth completion—the task of generating a dense, high-resolution depth map from sparse sensor data (such as LiDAR) guided by an RGB image—is a cornerstone of autonomous navigation and augmented reality. Traditionally, researchers have approached this as either a regression problem (predicting continuous values) or a classification problem (assigning pixels to discrete depth "bins").

While classification-based methods have shown promise, they suffer from a fundamental flaw: they rely on pre-defined, discrete depth categories shared across all pixels. This assumption that depth can be neatly categorized into fixed planes leads to "depth smearing," where the boundaries between foreground objects and backgrounds become blurry and artifact-ridden. As shown in the figure below, traditional classification methods struggle to represent the sharp transitions at object edges, creating "streaks" or "stripes."

Comparison between traditional classification-based methods and the proposed clustering-based method (CluDe).

The CluDe Framework

To bridge the gap between the stability of classification and the precision of continuous regression, we introduce CluDe (Clustering-based Depth completion). CluDe replaces fixed depth bins with learnable, scene-adaptive depth centers through a two-stage process:

  1. Clustering Transformer: Instead of using pre-defined depth bins, CluDe uses a transformer to learn a set of "depth centers" directly from the input data. These centers are feature vectors representing the most likely depth values for a specific scene, making the representation dynamic and adaptive.
  2. Hierarchical Translation Module: This core innovation addresses the remaining discreteness. CluDe estimates pixel-wise offsets that "nudge" the learned depth centers along the depth axis for every individual pixel. This transforms a discrete representation into a continuous one, allowing the model to represent any depth value within a range.
  3. Prune Translation Block: To handle noisy LiDAR data, CluDe includes a filtering mechanism that uses initial depth estimates to identify and prune outliers in the sparse input map, ensuring refinement is not misled by sensor errors.

Experimental Evaluation

We evaluated CluDe on two major benchmarks: the KITTI Depth Completion dataset (outdoor autonomous driving) and the VOID (Visual Odometry Indoor Dataset), which features varying levels of sparsity from 1500 points down to just 150 points per frame.

Key Results

  • State-of-the-Art Performance: CluDe achieved the top ranking on the VOID dataset, outperforming previous leaders like CostDCNet.
  • Boundary Fidelity: Qualitatively, CluDe produces significantly cleaner edges. By utilizing pixel-wise continuous representation, it effectively eliminates the "stripe" artifacts common in standard classification models.
  • Efficiency: On the KITTI test set, CluDe achieved a Mean Absolute Error (MAE) of 197.91 mm. It maintains a competitive latency of 0.14s per image, making it faster than many high-end regression models like NLSPN (0.22s).

Conclusion and Future Work

CluDe demonstrates that moving beyond fixed depth bins to a clustering-based, continuous representation is key to high-fidelity depth completion. By combining scene-adaptive centers with local offsets, the model achieves both convergence stability and edge precision.

One remaining challenge is the "long-tail" problem; in datasets like KITTI, pixels representing distances beyond 65 meters are rare, making it harder for the clustering mechanism to learn accurate representations for extreme distances. Future work will focus on improving performance in these sparse tail categories and further optimizing the transformer architecture for real-time deployment on edge devices.

Summary updated

Abstract

Depth completion is a long-standing challenge in computer vision, where classification-based methods have made tremendous progress in recent years. However, most existing classification-based methods rely on pre-defined pixel-shared and discrete depth values as depth categories. This representation fails to capture the continuous depth values that conform to the real depth distribution, leading to depth smearing in boundary regions. To address this issue, we revisit depth completion from the clustering perspective and propose a novel clustering-based framework called CluDe which focuses on learning the pixel-wise and continuous depth representation. The key idea of CluDe is to iteratively update the pixel-shared and discrete depth representation to its corresponding pixel-wise and continuous counterpart, driven by the real depth distribution. Specifically, CluDe first utilizes depth value clustering to learn a set of depth centers as the depth representation. While these depth centers are pixel-shared and discrete, they are more in line with the real depth distribution compared to pre-defined depth categories. Then, CluDe estimates offsets for these depth centers, enabling their dynamic adjustment along the depth axis of the depth distribution to generate the pixel-wise and continuous depth representation. Extensive experiments demonstrate that CluDe successfully reduces depth smearing around object boundaries by utilizing pixel-wise and continuous depth representation. Furthermore, CluDe achieves state-of-the-art performance on the VOID datasets and outperforms classification-based methods on the KITTI dataset.
BibTeX
@article{chen2024learning,
  author = {Shenglun Chen and Hong Zhang and Xinzhu Ma and Zhihui Wang and Haojie Li},
  title = {{Learning Pixel-wise Continuous Depth Representation via Clustering for Depth Completion}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2024},
  url = {https://openreview.net/forum?id=Z9M6YawbVz},
}
Figure