Image Guided Depth Upsampling using Anisotropic Total Generalized Variation

David Ferstl, Christian Reinbacher, Rene Ranftl, Matthias Rüther, Horst BischofarXiv 2026 · arXiv

TL;DR

This paper introduces a high-performance method for boosting the resolution of depth images by fusing low-res depth data with high-res intensity images. By using a sophisticated mathematical framework called Anisotropic Total Generalized Variation (TGV), the authors achieve state-of-the-art accuracy and real-time speeds, effectively turning low-cost, low-res depth sensors into high-fidelity 3D scanners.

What Problem Does This Paper Address?

Modern depth cameras, like the original Microsoft Kinect or specialized Time-of-Flight (ToF) sensors, are fantastic for capturing 3D information in real-time. However, they have a major "lateral resolution" problem: while they might give you a dense map of distances, the "image" itself is often tiny (e.g., 120x160 pixels) and noisy.

To use this data for high-quality object reconstruction or autonomous navigation, we need to "upsample" it—essentially zooming in and filling in the blanks. The challenge is doing this without blurring sharp edges (like the corner of a table) or creating "staircase" artifacts where smooth, slanted surfaces look like a series of flat steps. Traditional methods like bilateral filtering often struggle with these fine details or are too slow for real-time use.

Methods

The authors frame depth upsampling as a convex optimization problem. Their approach has three main pillars:

  1. Image Guidance via Anisotropic Diffusion: They don't just guess where the depth edges are; they use a high-resolution standard intensity (RGB) image as a guide. They calculate an "anisotropic diffusion tensor" from this image. This tensor tells the algorithm: "If there's a sharp edge in the photo, there's likely a sharp edge in the depth map, so allow a jump here. If the photo is smooth, keep the depth smooth too."
  2. Second-Order TGV Regularization: Most older methods use "Total Variation" (TV), which assumes the world is made of flat, front-facing blocks. This causes the "staircase" effect. By using second-order TGV, the authors allow the model to favor "piecewise affine" surfaces—basically, it understands that surfaces can be slanted.
  3. Primal-Dual Optimization: To solve this complex math quickly, they use a first-order primal-dual algorithm. This is highly parallelizable, meaning it can run very fast on modern hardware (GPUs), achieving multiple frames per second.
Figure 1. Upsampling of a low resolution depth image (a) using an additional high resolution intensity image (b) through image guided anisotropic Total Generalized Variation (c). Depth maps are color coded for better visualization.

Datasets

The researchers evaluated their work on two fronts:

  • Middlebury 2007 Datasets: A standard benchmark in computer vision (Art, Books, Moebius) where they simulated low-resolution and noise to compare against existing methods.
  • New Real-World Benchmark: The authors introduced a brand-new dataset using a real PMD Nano ToF camera paired with a CMOS intensity camera. They used a high-end structured light scanner to get "ground truth" (perfect) depth data for three scenes: Books, Shark, and Devil. This is significant because it accounts for real-world sensor noise, which is often more complex than simple mathematical simulations.

Key Insights and Ideas

  • Slants Matter: The move from first-order to second-order TGV is the "aha moment" for surface quality. It prevents the "flattening" of the world, allowing for natural-looking slanted planes.
  • Noise isn't Uniform: In real ToF sensors, noise increases as the reflected light (amplitude) decreases. The authors incorporated this into their "data term," weighting measurements based on their confidence. This makes the system much more robust to dark or non-reflective surfaces.
  • Directional Guidance: Unlike simple weighting, their anisotropic tensor actually orients the gradient direction during optimization. This ensures that depth boundaries align perfectly with visual boundaries in the high-res photo.
Visual comparison on the Middlebury 'Art' dataset. Note how the proposed method (f) preserves sharp edges and fine structures compared to others.

Key Results

  • Accuracy: The method consistently outperformed all state-of-the-art competitors (like Markov Random Fields or Non-local Means) across all magnification factors (x2 to x16) on the Middlebury datasets.
  • Speed: It is significantly faster than other high-quality methods. While some competitors took ~24 seconds per image, this method averaged 1.94 seconds for the same task. On real-world data with a x6.25 upsampling factor, it ran in roughly 318ms.
  • Real-World Performance: On the new real-world dataset, the method reduced the Root Mean Square Error (RMSE) by roughly 20-40% compared to standard interpolation and other guided filtering techniques.
Evaluation on real-world data (Books, Shark, Devil). The 'Error' column (d) shows how closely the upsampled result matches the high-precision ground truth.

Limitations and Future Work

The authors note a few hurdles:

  • Parallax/Occlusions: Because the depth sensor and the intensity camera are in slightly different positions, "occlusions" (areas one camera sees but the other doesn't) can cause errors near large depth jumps.
  • Sensor Artifacts: Extremely incorrect ToF measurements (due to multi-path interference or very low reflectivity) can still lead to displaced surfaces in the final result.

Future Work: The team plans to extend this framework to include temporal coherence. By looking at a sequence of frames rather than just one, they hope to use the "memory" of previous frames to reach even higher levels of accuracy and stability in video applications.

Summary updated

Abstract

In this work we present a novel method for the challenging problem of depth image upsampling. Modern depth cameras such as Kinect or Time of Flight cameras deliver dense, high quality depth measurements but are limited in their lateral resolution. To overcome this limitation we formulate a convex optimization problem using higher order regularization for depth image upsampling. In this optimization an anisotropic diffusion tensor, calculated from a high resolution intensity image, is used to guide the upsampling. We derive a numerical algorithm based on a primaldual formulation that is efficiently parallelized and runs at multiple frames per second. We show that this novel upsampling clearly outperforms state of the art approaches in terms of speed and accuracy on the widely used Middlebury 2007 datasets. Furthermore, we introduce novel datasets with highly accurate groundtruth, which, for the first time, enable to benchmark depth upsampling methods using real sensor data.
BibTeX
@misc{ferstl2026image,
  author = {David Ferstl and Christian Reinbacher and Rene Ranftl and Matthias Rüther and Horst Bischof},
  title = {{Image Guided Depth Upsampling using Anisotropic Total Generalized Variation}},
  year = {2026},
  eprint = {2611.03399},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03399},
}
Figure