Recovering 3D structure from 2D images is one of the most computationally demanding tasks in computer vision. Traditional stereo matching algorithms construct a high-resolution "cost volume"—a dense, multi-dimensional matrix representing the matching probability for every pixel across every possible depth level (disparity). Processing this volume requires massive memory bandwidth and intensive floating-point math, making it impractical for power-constrained edge devices like micro-drones, augmented reality (AR) glasses, and mobile robots.
To bring high-fidelity depth perception to low-power hardware, researchers have shifted away from brute-force calculation. Instead, they rely on hardware-software co-design: compressing the search space, streaming data to eliminate memory bottlenecks, and even offloading geometric computations directly onto physical camera optics.
Compressing the Cost Volume Bottleneck
The primary bottleneck in deep-learning-based stereo vision is the 4D cost volume, which is typically regularized using heavy 3D convolutions. To achieve real-time frame rates, modern architectures decouple spatial detail from the depth-search process.
Instead of building a massive, full-resolution cost volume, Sameh Khamis et al. demonstrated in StereoNet: Guided Hierarchical Refinement for Real-Time Edge-Aware Depth Prediction that a network can compute a very low-resolution cost volume to capture coarse geometric structures. Because depth boundaries inherently align with structural edges in the source image, the authors use lightweight 2D convolutions guided by the original high-resolution image to hierarchically upsample and sharpen the depth map. This approach achieves sub-pixel precision at a fraction of the computational cost.
[Left & Right Images] ---> Downsample ---> Low-Res Cost Volume (Coarse Depth)
|
[High-Res Left Image] ---> Edge Guidance ---> 2D Refinement ---> High-Res Depth Map
Other architectures eliminate 3D convolutions entirely. Hengli Wang et al. developed a self-supervised framework in PVStereo: Pyramid Voting Module for End-to-End Self-Supervised Stereo Matching that replaces heavy 3D convolutions with multi-scale cost volumes and recurrent updates, drastically lowering the instantaneous memory footprint. Taking this a step further, Jin Xie proved in BidNet: Binocular Image Dehazing Without Explicit Disparity Estimation that certain binocular tasks can bypass explicit disparity estimation altogether by directly learning cross-view correlations, avoiding the 4D volume entirely.
For hardware like AR glasses where even image rectification (aligning camera frames horizontally) introduces latency, Yongfan Liu et al. introduced hardware-optimized convolutions in Efficient Depth Estimation for Unstable Stereo Camera Systems on AR Glasses. By embedding homography-based positional encodings directly into the network, their system processes raw, misaligned sensor data without requiring a separate rectification step, cutting end-to-end latency nearly in half.
Exploiting Contextual and Temporal Priors
A pixel in a stereo pair does not need to be compared against every possible disparity if its spatial or temporal context is already known. Restricting the search space allows algorithms to drop their computational overhead significantly.
In autonomous driving contexts, the underlying geometry of the environment is highly predictable. Han Ma et al. leveraged this in Multiple Lane Detection Algorithm Based on Optimised Dense Disparity Map Estimation by using the road geometry extracted from the previous video frame to dynamically narrow down the disparity search range in the current frame, reducing processing time by 37%.
In video-based depth sensing, temporal redundancy offers an even larger optimization window. The authors of Stereo Matching in Time: 100+ FPS Video Stereo Matching for Extended Reality unrolled iterative cost aggregation across the temporal dimension. By warping features from the previous frame into the current frame based on camera motion, the system reuses historical computations rather than recalculating depth from scratch, enabling frame rates exceeding 100 FPS on mobile hardware.
Frame t-1: [Compute Features] ---> [Aggregate Cost] ---> [Depth Map t-1]
|
Warp Features (via Motion Vector)
|
v
Frame t: [Compute Features] ---> [Temporal Fusion] ---> [Depth Map t]
When continuous stereo matching is too expensive, hybrid pipelines can bridge the gap. Simone Gasparini et al. demonstrated in Stereo Camera Tracking for Mobile Devices that a system can restrict heavy stereo processing to initial depth reconstruction and periodic updates, relying on lightweight monocular tracking for intermediate frames to preserve battery life.
Streaming Pipelines and Bitwise Logic on Silicon
When deploying depth sensing on custom silicon like Field Programmable Gate Arrays (FPGAs), the challenge shifts from minimizing operations to optimizing memory access patterns. Standard processors waste energy moving data back and forth between the CPU and RAM. FPGAs solve this by using fully pipelined, streaming architectures.
In R$^3$SGM: Real-time Raster-Respecting Semi-Global Matching for Power-Constrained Systems, Oscar Rahnama et al. redesigned the classic Semi-Global Matching (SGM) algorithm for single-pass streaming. Instead of storing the entire image cost matrix in memory to calculate multi-directional path aggregations, their architecture processes pixels sequentially as they arrive from the sensor raster. This streaming approach allows a hybrid FPGA-CPU SoC to deliver dense, high-accuracy depth maps at over 50 FPS within a strict 5-Watt power envelope, as further detailed by Oscar Rahnama et al. in Real-Time Highly Accurate Dense Depth on a Power Budget using an FPGA-CPU Hybrid SoC.
Silicon efficiency can be maximized by replacing floating-point math with integer-based logic. Jasmine Banks et al. showed in Fast and Robust Stereo Matching Algorithms for Mining Automation that non-parametric transforms, such as the Census transform, map local pixel intensities into binary strings. Matching costs can then be calculated using simple bitwise XOR operations and bit-counting (Hamming distance), which execute natively on hardware in a single clock cycle.
To reduce the raw volume of data entering the pipeline, Charan Kumar Vala et al. integrated Discrete Wavelet Transforms (DWT) into the disparity pipeline in High-Speed Low-Complexity Guided Image Filtering-Based Disparity Estimation. Processing images in the wavelet domain cuts the data workload by 50% and enables a fully pipelined architecture that outputs a new depth estimate on every single clock cycle.
Shifting the Burden to Physical Optics
The most radical efficiency gains come from eliminating the digital matching problem entirely by offloading geometric calculations to the camera's optical hardware.
This direction builds on the foundational insight by Alex Pentland in A New Sense for Depth of Field, which demonstrated that the inherent blur of an optical system—the focal gradient—contains direct geometric data about depth. By measuring edge blur or local frequency changes, depth can be recovered without searching for corresponding pixels across two views.
Modern hardware-software co-design takes this concept to its logical extreme. Junjie Luo et al. introduced a method called Coupled Optical Differentiation in Depth from Coupled Optical Differentiation. By capturing images with slight, structured variations in lens power and aperture size, the physical optics perform a mathematical differentiation on the incoming light. Because the geometry is pre-computed by the lens itself, the digital processor only needs to execute a simple, closed-form formula requiring just 36 floating-point operations (FLOPs) per pixel.
Incoming Light ---> [Physically Modulated Lens] ---> Differentiated Images ---> [36 FLOPs/Pixel Formula] ---> Dense Depth
To achieve this in a single snapshot without moving parts, Junjie Luo 0009 et al. designed a bio-inspired handheld camera in Focal Split: Untethered Snapshot Depth from Differential Defocus. Mimicking the multi-layered retina of a jumping spider, the system splits incoming light onto different focal planes simultaneously. Using a mathematically rigorous derivative formula, it extracts dense depth maps using only 500 FLOPs per pixel, enabling real-time, untethered depth sensing on a standard Raspberry Pi under a 5-Watt power budget.
Key Foundational Work
Relative depth from vergence micromovements (1993) — contributed 1 concept elements to this direction.
A Parallel Matching Algorithm for Stereo Vision (1985) — contributed 1 concept elements to this direction.
A New Sense for Depth of Field (1985) — contributed 1 concept elements to this direction.