Defocus Blur Synthesis and Deblurring via Interpolation and Extrapolation in Latent Space

Ioana Mazilu, Shunxin Wang, Sven Dummer, Raymond N. J. Veldhuis, Christoph Brune, Nicola Strisciuglio

Introduction

In high-resolution microscopy, capturing three-dimensional biological structures in perfect focus is a significant technical challenge. Because cells and tissues occupy different focal planes, a single snapshot often results in a mix of sharp and blurry regions. Traditional computational approaches to deblurring typically rely on complex, resource-intensive architectures like CycleGANs or multi-scale U-Nets. While these models can produce high-quality results, they are often computationally expensive and struggle to generalize to blur levels not explicitly encountered during training.

This research proposes a more flexible, lightweight alternative that treats defocus blur as a continuous spectrum rather than a binary state. By enforcing a linear structure within the latent space of an autoencoder, the researchers demonstrate that both blur synthesis and deblurring can be achieved through simple geometric operations: interpolation and extrapolation.

Methodology: Regularizing the Latent Space

The foundation of this approach is a Regularized Autoencoder. In a standard autoencoder, an image is compressed into a low-dimensional latent space and then reconstructed. However, these latent representations are typically disorganized, making it difficult to manipulate specific image attributes like blur in a predictable way.

To address this, the authors introduced two forms of regularization designed to force the latent space into a linear configuration:

  1. Indirect Regularization: The model is trained to reconstruct an intermediate blurry image ($x_b$) using a latent code generated by interpolating between the codes of a sharp image ($x_a$) and a heavily blurred image ($x_c$).
  2. Direct Regularization: An explicit penalty is added to the loss function to minimize the mathematical distance between the interpolated latent code and the actual encoded representation of the intermediate image.

By enforcing this linearity, the model learns that "moving" along a specific vector in the latent space corresponds directly to changing the blur level.

  • Blur Synthesis: To generate a blurrier version of an image, the model interpolates between a sharp and a blurry latent code.
  • Deblurring: To sharpen an image, the model performs extrapolation—moving further along the established line in the direction of decreasing blur, potentially reaching a "zero-blur" state that exceeds the sharpness of the training samples.

Experimental Setup and Datasets

The study utilized the BBBC006v1 collection from the Broad Bioimage Benchmark Collection. This dataset consists of 384 cell slides stained for nuclei and cell structure. Each slide was captured across a "z-stack" of 34 different focal lengths. The researchers focused on images captured above the optimal focal plane, using even-numbered z-stack levels to capture significant visual variation for training, validation, and testing.

Key Results and Insights

Latent Space Geometry

The effectiveness of the regularization was measured using the Linear Dependence Score (LDS). The results showed that direct regularization significantly organized the latent space, increasing the LDS from a baseline of 0.59 to 0.76. While direct regularization provided the most organized geometry, the authors noted a slight "fidelity trade-off," where extreme enforcement of geometry can lead to minor decreases in reconstruction quality.

Synthesis and Deblurring Performance

In blur synthesis tasks, the regularized models outperformed baselines by eliminating "ghosting" or blending artifacts. While standard models often produce a ghostly overlay of two images during interpolation, the proposed model produces a natural, continuous transition of focus.

For deblurring, the model achieved competitive results using the Peak Signal-to-Noise Ratio (PSNR) metric. Specifically, for cell structure images (w2), the direct regularization model achieved a PSNR of 24.46 dB when comparing extrapolated (deblurred) images to the ground truth sharp images.

Discussion and Future Work

Despite the success of the linear model, PCA projections of the latent space revealed that the true trajectory of blur is often "curvilinear" rather than perfectly straight. This suggests that the relationship between image content and defocus blur is highly complex.

Future research directions include:

  • Representation Disentanglement: Separating the "content" of the cell from the "blur level" in the latent space to allow for more precise manipulation without affecting biological features.
  • Single-Image Deblurring: Currently, the model performs best when provided with two images of the same slide at different blur levels to establish a trajectory. Developing methods to determine this direction from a single image remains a primary goal.

By treating blur as a navigation problem within a structured mathematical space, this method offers a promising, efficient path toward high-fidelity microscopy imaging.

Summary updated

Abstract

Though modern microscopes have an autofocusing system to ensure optimal focus, out-of-focus images can still occur when cells within the medium are not all in the same focal plane, affecting the image quality for medical diagnosis and analysis of diseases. We propose a method that can deblur images as well as synthesize defocus blur. We train autoencoders with implicit and explicit regularization techniques to enforce linearity relations among the representations of different blur levels in the latent space. This allows for the exploration of different blur levels of an object by linearly interpolating/extrapolating the latent representations of images taken at different focal planes. Compared to existing works, we use a simple architecture to synthesize images with flexible blur levels, leveraging the linear latent space. Our regularized autoencoders can effectively mimic blur and deblur, increasing data variety as a data augmentation technique and improving the quality of microscopic images, which would be beneficial for further processing and analysis. The code is available at https://github.com/nis-research/linear-latent-blur .
BibTeX
@inproceedings{mazilu2023defocus,
  author = {Ioana Mazilu and Shunxin Wang and Sven Dummer and Raymond N. J. Veldhuis and Christoph Brune and Nicola Strisciuglio},
  title = {{Defocus Blur Synthesis and Deblurring via Interpolation and Extrapolation in Latent Space}},
  booktitle = {CAIP},
  year = {2023},
  url = {https://openreview.net/forum?id=MKC3JvOIITk},
}
Figure