Overcoming the Staircase Effect in Discrete Depth Representations

62 concepts57 papers1 proposal

When we look at the physical world, distance is a smooth, unbroken continuum. Yet, many of the most successful algorithms in computer vision—such as plane-sweeping stereo, multiplane images (MPIs), and cost-volume optimizations—approximate this continuum by slicing 3D space into a finite stack of discrete layers or disparity bins.

While computationally convenient, this discretization introduces a fundamental geometric conflict. When a continuous, slanted surface is forced into a rigid set of parallel depth planes, it cannot be represented smoothly. Instead, it fractures into a series of jagged, step-like structures. This phenomenon is known as the staircase effect or depth quantization artifact. Understanding why these artifacts occur and how modern techniques resolve them is essential for high-fidelity 3D reconstruction and novel view synthesis.


The Geometry of the Staircase Artifact

To build an intuition for why discrete depth representations fail, imagine a camera looking down a long, straight hallway. The walls and floor recede continuously into the distance.

If we use a traditional plane-sweeping algorithm or a multiplane image framework, we place a fixed number of virtual "glass panes" at incremental distances from the camera. Every point in the scene must be projected onto one of these panes.

Continuous Sloped Surface:   /
                            /
                           /
                          /

Discrete Approximation:     _│
                         _│
                       _│
                     _│

This quantization creates three distinct types of visual and geometric failures:

  1. The Oblique Angle Breakdown: As early work by Nelson Max et al. in Hierarchical Image-Based Rendering using Texture Mapping Hardware demonstrated, low-precision depth channels create visible gaps and discontinuities that become highly pronounced when the viewing angle is nearly parallel to the surface. When a surface is viewed obliquely, a single smooth slope spans multiple depth layers, forcing the rendering pipeline to jump abruptly from one plane to the next.
  2. The Inter-Plane Void: Real-world geometry rarely aligns perfectly with predefined virtual planes. As observed by Hideo Saito et al. in Real-time free viewpoint video from uncalibrated cameras using plane-sweep algorithm, objects or parts of objects that reside between the sampled layers cannot be accurately localized. When rendering novel views, this sampling frequency constraint manifests as severe ghosting, aliasing, or visual fracturing.
  3. The Layering and Sheet Effect: In complex reconstructions, such as those utilizing multiplane representations or forward-facing novel view synthesis, the scene can begin to look like a series of flat, disconnected cardboard cutouts. Amandeep Kumar et al. showed in Generative Multiplane Neural Radiance for 3D-Aware Image Generation that this layering or ghosting becomes particularly problematic at shallow angles, where the lack of a continuous volumetric representation causes the underlying geometry to lose its structural fluidity.

The Optimization Bias Behind the Steps

Why do algorithms default to these jagged steps rather than smoothing them out? The root cause lies in how traditional stereo matching and depth estimation frame the optimization problem.

When depth estimation is treated as a discrete labeling problem within a cost volume, the algorithm calculates a matching cost for each pixel across a fixed set of disparity or depth hypotheses. In Semi-Global Stereo Matching with Surface Orientation Priors, Daniel Scharstein explained that standard optimization objectives inherently favor assigning identical, discrete depth labels to neighboring pixels over modeling continuous gradients. Because the mathematical objective penalizes changes in labels to enforce smoothness, it inadvertently forces slanted or receding surfaces into flat, step-like plateaus.


Moving Beyond Fixed Slices

To eliminate the staircase effect, researchers have developed mechanisms that bridge the gap between discrete computational structures and the continuous physical world. These solutions generally fall into three categories: sub-pixel refinement, adaptive geometry, and implicit coordinate fields.

1. Sub-Pixel Offsets and Continuous Loss Functions

Instead of forcing a pixel to choose a single discrete depth bin, modern architectures predict continuous adjustments relative to those bins.

For instance, Divyansh Garg et al. introduced a Continuous Disparity Network in Wasserstein Distances for Stereo Disparity Estimation. Instead of relying on the expected value of fixed bins—which introduces discretization errors—their approach predicts per-bin continuous offsets and optimizes the distribution using a Wasserstein distance loss. This allows the network to resolve precise sub-pixel shifts, smoothing out the harsh boundaries typically found at object edges. Similarly, in ultra-narrow baseline scenarios like smartphone dual-pixel sensors, Minjun Kang et al. demonstrated in Facial Depth and Normal Estimation Using Single Dual-Pixel Camera that phase-shift interpolation can resolve sub-pixel granularities, turning a highly constrained discrete search space into a fluid depth map.

2. Adaptive and Posed Geometry

If rigid, parallel planes cause artifacts on slanted surfaces, an intuitive alternative is to let the planes tilt and conform to the scene's actual layout.

Fixed Slices:       │  │  │  │        Adaptive Slices:    /  /  /  /
(Causes steps)      │  │  │  │        (Aligns to slope)  /  /  /  /

The authors of Structural Multiplane Image: Bridging Neural View Synthesis and 3D Reconstruction, Mingfang Zhang et al., replaced the traditional stack of rigid, camera-parallel slices with adaptive, posed planes. By dynamically aligning the orientation of the representation's layers with the underlying structural geometry of the scene, they prevented slanted surfaces from being sliced into multiple discrete layers, effectively neutralizing the staircase pattern and preserving texture clarity.

3. Neural Implicit Fields

The most radical departure from discretization is abandoning grids and planes altogether, treating depth as a continuous function of spatial coordinates.

In InfiniDepth: Arbitrary-Resolution and Fine-Grained Depth Estimation with Neural Implicit Fields, Hao Yu et al. shifted from discrete 2D image grids to neural implicit fields. Because an implicit field evaluates depth as a continuous function, it completely bypasses the resolution limits of pixel grids and depth bins. This allows the system to reconstruct fine-grained, intricate structures—such as thin wires, leaves, and sharp edges—that would otherwise fall into the voids between discrete sampling intervals.

By evolving from rigid, quantized slices to continuous, geometry-aware representations, modern computer vision pipelines successfully eliminate the staircase effect, achieving the fluid realism demanded by advanced 3D reconstruction and novel view synthesis.

Key Foundational Work

Hierarchical Image-Based Rendering using Texture Mapping Hardware (1999) — contributed 1 concept elements to this direction.
Range Estimation From Focus Using a Non-frontal Imaging Camera (1993) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_depth_discretization_artifacts_fc1a1669,
  title = {depth discretization artifacts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/depth-discretization-artifacts/rdr_fc1a1669fd1145fd80bd83e39f1a52c0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_depth_discretization_artifacts_fc1a1669_v1,
  title = {depth discretization artifacts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/depth-discretization-artifacts/rdr_fc1a1669fd1145fd80bd83e39f1a52c0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake