Frequency-Aware Deep Depth from Focus

Tao Yan, Yingying Wang, Jiangfeng Zhang, Yuhua Qian, Jieru Jia, Lu Chen, Feijiang Li

Abstract

Depth from Focus (DFF) is a critical technique for 3D reconstruction, yet traditional spatial-domain convolutional neural networks (CNNs) often suffer from detail loss and blurring. We present the Frequency-Aware Deep Depth from Focus (FAD) network, a novel architecture that integrates global frequency-domain information via Fourier transforms into a deep learning pipeline. By combining local spatial features with global structural insights, FAD achieves state-of-the-art accuracy and superior detail preservation across synthetic and real-world datasets.

Introduction

Reconstructing 3D geometry from a sequence of images with varying focus—known as Depth from Focus (DFF)—is essential for high-resolution imaging and microscopy. However, modern DFF methods relying solely on spatial CNNs struggle with "near-sightedness." Because convolutions are inherently local operations, they often fail to maintain global structural consistency, leading to blurred edges and smoothed-over textures. This is particularly problematic in texture-less regions where local focus cues are weak and difficult to detect.

Methodology: The FAD Architecture

The Frequency-Aware Deep Depth from Focus (FAD) model overcomes these limitations through a hybrid spatial-frequency approach. The core innovation is the Time-Frequency Joint Module, which operates alongside standard spatial feature extraction within a U-Net-style backbone.

The FAD network architecture, highlighting the Time-Frequency Joint Module and the integration of FFT for global feature extraction.

Key Components:

  • 3D Convolutions: FAD employs 3D convolutions to process the input focal stack, capturing how focus evolves across the depth dimension (the temporal/focal axis).
  • Clarity Area Identification (CAI): This specialized module targets subtle defocus variations, even in areas with weak texture.
  • Frequency Domain Feature Extraction: By applying a Fast Fourier Transform (FFT) to latent features, the model gains a "mathematical bird's-eye view" of the scene. An adaptive weight matrix filters these frequencies to enhance relevant focus information before an inverse FFT (iFFT) returns the data to the spatial domain.
  • Multi-Scale Weighted Loss: Training utilizes a pyramid-like loss function to ensure consistency across four different scales, from coarse shapes to fine details.

Experiments and Results

The FAD model was rigorously tested on a diverse set of benchmarks, including synthetic datasets (SLFD, Pov-Ray, 4D Light Field) and real-world datasets (NYU Depth V2, DDFF 12-Scene). It was also applied to microscopic samples through a pseudo-labeling fine-tuning process.

Qualitative results across various datasets and quantitative comparison of Average RMSE, showing FAD's superior performance.

FAD consistently outperforms existing state-of-the-art (SOTA) methods like DfFW and AiF. Quantitatively, on the SLFD dataset, FAD achieved a Mean Squared Error (MSE) of 0.3460, a significant improvement over the previous best of 0.8939. Qualitatively, the model excels at reconstructing complex geometries—such as thin structures and surface scratches—that traditional models typically smooth over. Despite the added complexity of FFTs, the model remains efficient with an inference time of approximately 0.12 seconds per stack.

Conclusion

The FAD network demonstrates that integrating frequency-domain information is a highly effective strategy for high-fidelity 3D reconstruction. By balancing local spatial features with global frequency insights, the model sets a new standard for Depth from Focus. Future work will explore multi-task learning to develop a universal DFF model capable of zero-shot generalization across diverse optical systems.

Summary updated

Abstract

In large aperture imaging, the shallow depth of field (DoF) phenomenon requires capturing multiple images at different focal levels, allowing us to infer depth information using depth from focus (DFF) techniques. However, most previous works design convolutional neural networks from a time domain perspective, often leading to blurred fine details in depth estimation. In this work, we propose a frequency-aware deep DFF network (FAD) that couples multi-scale spatial domain local features with frequency domain global structural features. Our main innovations include two key points: First, we introduce a frequency domain feature extraction module that uses the Fourier transform to transfer latent focus features into the frequency domain. This module adaptively captures essential frequency information for focus changes through element-wise multiplication, enhancing fine details in depth results while preserving global structural integrity. Second, the time-frequency joint module of FAD improves the consistency of depth information in sparse texture regions and the continuity in transition areas from both local and global complementary perspectives. Comprehensive experiments demonstrate that our model achieves compelling generalization and state-of-the-art depth prediction across various datasets. Additionally, it can be quickly adapted to real-world applications as a pre-trained model.
BibTeX
@inproceedings{yan2025frequency,
  author = {Tao Yan and Yingying Wang and Jiangfeng Zhang and Yuhua Qian and Jieru Jia and Lu Chen and Feijiang Li},
  title = {{Frequency-Aware Deep Depth from Focus}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/241},
}
Figure