When we think about estimating depth from a 2D image, our natural instinct is to treat it as a regression problem. After all, distance is a continuous physical quantity. A pixel could be 1.42 meters or 50.38 meters away from the camera. For years, standard computer vision architectures followed this intuition, training deep networks to output a single continuous scalar per pixel by minimizing distance-based loss functions like Mean Squared Error (L2) or Mean Absolute Error (L1).
However, direct continuous regression suffers from fundamental flaws that limit its accuracy, training stability, and ability to handle complex scene geometries. To overcome these limitations, a powerful paradigm shift has emerged: reformulating continuous depth estimation as a discrete classification task over quantized depth intervals or "bins."
The Regression Trap: Blurry Edges and Depth Averaging
To understand why classification works so well, we must first understand why continuous regression fails. The core issue lies in how regression models handle uncertainty and depth discontinuities.
Imagine a pixel located precisely on the sharp silhouette of a foreground object—such as a tree branch—with a distant mountain in the background. The tree is 2 meters away, while the mountain is 2,000 meters away. Because of slight misalignments in training data or spatial ambiguities in the image, the network may be uncertain about which object the pixel belongs to.
If the network is trained using a standard regression loss, it minimizes its expected error by splitting the difference and predicting the mathematical average—roughly 1,001 meters. But there is absolutely nothing at 1,001 meters. This phenomenon, known as the "depth-averaging" artifact, results in physically impossible predictions and visually smeared, blurry object boundaries. As Gengshan Yang notes in Inferring Distributions Over Depth from a Single Image, continuous regression acts as an unfortunate regularizer that forces models to output the mean of conflicting likely values.
Continuous Regression (L2 Loss):
True Depth A (2m) <------------------ Predicted Mean (1001m) ------------------> True Depth B (2000m)
(A blurry, physically empty artifact)
Discrete Classification:
[ Bin 1: 2m ] --> Probability: 0.5
[ Bin 2: 10m ] --> Probability: 0.0
...
[ Bin N: 2000m] -> Probability: 0.5
(Preserves both distinct hypotheses)
By contrast, treating depth estimation as a classification task over discrete bins allows the network to output a probability distribution across multiple depth hypotheses. As Thomas Neff demonstrates in DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks, classification-based surface localization completely avoids these depth-averaging artifacts. Instead of bleeding foreground and background depths together, the model places samples cleanly on either the foreground or background surface, preserving crisp geometric boundaries.
Simplifying the Optimization Landscape
Beyond fixing edge artifacts, discretization fundamentally changes the optimization problem, making it significantly easier for deep networks to learn.
Mapping raw pixels directly to an infinite, continuous scalar space forces a network to solve a highly non-linear regression problem. In contrast, identifying whether a pixel falls within a specific range of distances is a much simpler verification task. Yuanzhouhan Cao et al. first demonstrated this advantage in Estimating Depth from Monocular Images as Classification Using Deep Fully Convolutional Residual Networks, showing that classification reduces the computational burden on network parameters and naturally provides a pixel-wise measure of confidence. This representational strategy simplifies optimization even when adapting massive, frozen vision foundation models to specialized domains, as shown by Beilei Cui et al. in Surgical-DINO: Adapter Learning of Foundation Models for Depth Estimation in Endoscopic Surgery.
Discretization also provides substantial benefits for training dynamics:
- Stable Gradients: As argued by Yafei Song et al. in Single Image Dehazing Using Ranking Convolutional Neural Network, predicting discrete categories provides more stable gradients and accelerates model convergence compared to unconstrained numerical regression.
- Cross-Domain Robustness: In Deep attention-based classification network for robust depth prediction, Ke Xian et al. reveal that classification introduces "easy negatives." When a model trains on diverse environments (e.g., indoor rooms vs. wide highways), it can activate only the relevant depth bins for that specific domain. This prevents training data from one environment from corrupting or interfering with the logic of another.
Incorporating Spatial Geometry: Ordinal and Binary Binning
Standard classification frameworks (like predicting whether an image contains a cat, dog, or car) treat all misclassifications equally. If the true label is "cat," predicting "dog" is just as wrong as predicting "car."
Depth, however, possesses an inherent spatial ordering. If a pixel's true depth is 5 meters, predicting 6 meters is a minor error, whereas predicting 50 meters is catastrophic. To respect this geometry, researchers use specialized binning and loss formulations.
Spacing-Increasing Discretization (SID)
In natural scenes, our visual system requires high precision for nearby objects but tolerates lower precision for distant structures. Uniformly dividing depth into equal intervals (e.g., every 1 meter) wastes bins on the distant background where fine-grained distinctions are impossible to make from a single image.
To address this, Huan Fu et al. introduced a "spacing-increasing discretization" (SID) strategy in their seminal paper, Deep Ordinal Regression Network for Monocular Depth Estimation (DORN). SID quantizes depth using a logarithmic scale, creating narrow, high-resolution bins for nearby spaces and progressively wider bins for long-range distances.
Uniform Binning:
| 1m | 1m | 1m | 1m | 1m | 1m | 1m | 1m |
Spacing-Increasing Discretization (Logarithmic):
| 0.5m | 1m | 2m | 4m | 8m |
By framing depth estimation as an ordinal regression problem over these logarithmic intervals, the network learns the sequential relationships between bins. As validated by Bo Li et al. in Monocular Depth Estimation with Hierarchical Fusion of Dilated CNNs and Soft-Weighted-Sum Inference, this logarithmic approach naturally accommodates the long-tail distribution of distant points in natural scenes, making the model highly robust against rare, far-away outliers that typically derail standard regression networks.
Binary Planes and Implicit Queries
The classification paradigm can be compressed to its logical extreme: transforming depth estimation into a sequence of simple binary decisions.
In Bi3D: Stereo Depth Estimation via Binary Classifications, Abhishek Badki et al. reframe stereo depth by asking a single question relative to a reference plane: Is this object in front of or behind this specific depth? This binary formulation allows users to trade computational latency for accuracy in real-time systems by querying only the specific depth planes relevant to an application.
Similarly, J.S. Watson et al. leverage this concept for augmented reality in Virtual Occlusions Through Implicit Depth. By training an implicit query network to classify whether a real-world surface sits in front of or behind a virtual object, they bypass continuous depth estimation entirely, yielding exceptionally sharp occlusion boundaries and stable spatial ordering near complex edges.
Managing Ambiguity and Multi-Modal Distributions
Real-world scenes are full of visual ambiguities: semi-transparent glass tables, glossy reflections on a floor, or fine hair strands. In these scenarios, a single pixel does not just map to one depth; it physically contains light reflecting from multiple distinct distances simultaneously.
Continuous regression cannot represent this reality because it is mathematically constrained to output a single scalar. Discretized classification, however, excels here. In Towards Multimodal Depth Estimation from Light Fields, Titus Leistner et al. show that outputting a probability distribution across discrete bins allows a model to capture multi-peaked (multi-modal) distributions. If a pixel captures a reflection on a window pane, the network can assign high probabilities to both the nearby glass and the distant object reflected within it.
Furthermore, shifting to a classification framework unlocks advanced loss functions. As Yikang Ding et al. demonstrate in Enhancing Multi-view Stereo with Contrastive Matching and Weighted Focal Loss, treating depth planes as discrete hypotheses allows the integration of classification-specific losses like Focal Loss. This enables the network to dynamically down-weight easy, uniform regions (like flat walls) and focus its capacity on difficult, highly ambiguous regions that distance-based losses often ignore.
To convert these discrete probabilities back into smooth, continuous depth values during inference without reintroducing regression artifacts, models often employ a soft-argmax operation. As utilized by Jiahao Lin in HDNet: Human Depth Estimation for Multi-person Camera-Space Localization, taking a soft-weighted sum over the discrete log-depth bins provides a highly stable training signal while allowing sub-bin precision for continuous localization.
Hybrid Architectures: Coarse Binning with Fine Residuals
While pure classification solves the structural and optimization issues of regression, it introduces a new challenge: quantization error. If a scene contains 100 depth bins, the model's raw categorical prediction cannot distinguish between an object at the front edge of a bin versus the back edge of that same bin.
To achieve both the structural sharpness of classification and the millimeter precision of regression, modern architectures combine them into a hybrid framework: coarse-to-fine plane-residual estimation.
Hybrid Depth Prediction:
Step 1: Classification -> Identify the coarse depth plane (e.g., Bin 3: 5.0m)
Step 2: Regression -> Predict the continuous residual offset (e.g., +0.12m)
Final Metric Depth = 5.12m
In Depth Completion Using Plane-Residual Representation, Byeong-Uk Lee et al. factorize depth estimation into these two distinct sub-tasks. The network first runs a discrete classification to locate the coarse depth plane a pixel belongs to, and then executes a continuous regression to determine the exact residual distance from that plane.
This factorization radically simplifies the optimization landscape. The classification step handles the macro-geometry and isolates depth discontinuities, ensuring that object boundaries remain perfectly sharp. The regression step is left with a much simpler task: modeling tiny local variations within a strictly bounded range. Kratarth Goel et al. successfully deployed this hybrid strategy in QuadroNet: Multi-Task Learning for Real-Time Semantic Depth Aware Instance Segmentation, demonstrating that it captures broad depth ranges via quantization while preserving the fine-grained metric details necessary for identifying subtle road textures and nearby obstacles in autonomous driving.
Conclusion
By treating depth estimation as a discrete classification problem, the computer vision community bypassed the mathematical traps of continuous regression. Discretization prevents depth-averaging artifacts at object boundaries, stabilizes network gradients, and provides a natural mechanism for representing spatial uncertainty and multi-modal scenes. Whether implemented as pure ordinal classification, binary plane queries, or hybrid plane-residual networks, this paradigm shift remains a cornerstone of robust, sharp, and real-time 3D scene understanding.
Key Foundational Work
Pixel-level encoding and depth layering for instance-level semantic segmentation (2016) — contributed 2 concept elements to this direction.
Pixel-level Encoding and Depth Layering for Instance-level Semantic Labeling (2016) — contributed 2 concept elements to this direction.
Estimating Depth from Monocular Images as Classification Using Deep Fully Convolutional Residual Networks (2016) — contributed 1 concept elements to this direction.