TL;DR
OmniLens++ tackles the challenging task of "blind" lens aberration correction—removing spatially-varying optical blur without knowing the physical lens specifications or Point Spread Functions (PSFs) beforehand. By building a massive, uniformly sampled synthetic lens library (AODLibpro) and introducing a Latent PSF Representation (LPR) that predicts optical degradation priors directly from degraded images, OmniLens++ achieves state-of-the-art zero-shot restoration across diverse real-world and minimalist optical systems.
What Problem Does This Paper Address?
All optical systems suffer from aberrations—spatially-varying blur and chromatic fringing caused by manufacturing compromises, alignment errors, or ultrathin/minimalist lens designs (e.g., in smartphones or AR/VR hardware).
Existing solutions face a dilemma:
- Non-blind methods require laborious calibration to measure the lens's Point Spread Function (PSF), making them impractical for end-users.
- Blind methods using pre-trained "Lens Libraries" (LensLib-PT), such as the prior OmniLens, struggle with two bottlenecks:
- Limited data scalability: Synthetic lens datasets like AODLib-EAOD have biased distributions because they sample lenses based on simple optical design metrics (e.g., average RMS spot radius) that fail to capture real, spatially-varying blur severity.
- Lack of degradation guidance: Without access to ground-truth PSFs at test time, deep networks lack an explicit prior to adaptively handle wildly different aberration profiles.
Methods
Stage I: Latent PSF Representation (LPR)
[ PSF Map (SFR + FWHM) ] ──> [ PSF-VQVAE Encoder ] ──> [ Vector Quantization Codebook ] ──> Latent Prior
│ ▲
└──> [ PSF-Conditioned Reg. ] ──┘ (Models forward blur)
Stage II: Foundational Aberration Correction (FoundCAC)
[ Degraded Image ] ──> [ Degradation Encoder ] ──> [ Feature Matching / Codebook ] ──> Latent PSF Prior
│ │
└─────────────> [ Image Encoder ] ───────> [ Deep Fusion (RSTB) ] <──────────────────┘
│
└──> [ Decoder ] ──> [ Restored Image ]
OmniLens++ introduces two primary innovations spanning data generation and model architecture:
- AODLibpro (Scalable Lens Library Generation):
- Expanded Specifications: Incorporates aspheric lens surfaces (higher-order polynomial profiles) and depth-of-field image plane perturbations into evolutionary automatic optical design (EAOD).
- Hybrid Sampling Basis: Instead of relying on raw optical ray-tracing statistics, the authors quantify optical degradation via an Optical Image Quality (OIQ) metric—combining PSNR, SSIM, and Spatial Frequency Response (SFR) over 5 Field-of-View (FoV) positions. Lenses are categorized into 3 Severity-Classes and 6 Spatial-Classes (e.g., center-blurred vs. edge-blurred vs. uniform), ensuring uniform dataset coverage across 18 distinct degradation archetypes.
- Latent PSF Representation (LPR) & FoundCAC:
- Stage I (Pre-training LPR): PSFs are compressed into 2D maps based on radial SFR curves and Full Width at Half Maximum (FWHM). A discrete VQ-VAE learns a discrete codebook of these PSF features, constrained by a forward degradation regularizer that simulates how these PSF latents blur sharp images.
- Stage II (FoundCAC Model): A blind dual-encoder UNet. One branch extracts image content, while a degradation encoder queries the frozen LPR codebook to infer the latent PSF characteristics solely from the blurry image. The image features and retrieved latent PSF priors are fused using Residual Swin Transformer Blocks (RSTB).
Datasets
- AODLibpro: 3,600 automatically designed training lenses and 54 unseen test lenses evenly split across 18 degradation sub-classes (generated using the DeepLens simulator on Flickr2K images).
- RealLens-Sim: Synthetic benchmark containing 4 minimalist optical systems (MOS, spherical and aspheric), 1 metalens, a smartphone lens across 3 misalignment error levels, and 2 commercial high-end lenses.
- RealLens-Snap: Real-world captured photos from custom-fabricated single lenses, a commercial CAY-A 50mm f/1.4 minimalist lens, Canon 24mm f/1.4, Sony 18–135mm DSLRs, and nano-optics (metalens) data.
Key Insights and Ideas
- Image-domain metrics beat optical design indicators for dataset balancing: Standard optical metrics like RMS spot radius do not linearly correlate with human-perceived image blur or FoV variation trends. Sampling by image quality (OIQ) ensures training diversity that genuinely unlocks data scaling.
- PSF guidance without PSFs at inference: While direct end-to-end PSF regression from a single image is severely ill-posed, retrieving quantized PSF embeddings from a regularized VQ-VAE codebook provides a well-constrained proxy that acts as an effective degradation prior.
- Chromatic aberration doesn't require explicit sampling: Analysis showed model performance was invariant to chromatic aberration severity in sampling. Because unconstrained optical optimization naturally generates chromatic fringing, the network learns to correct purple fringing without dedicated stratification.
Key Results
- RealLens-Sim Benchmark: OmniLens++ (FoundCAC) achieves 26.66 dB PSNR / 0.851 SSIM / 0.1642 LPIPS, outperforming previous state-of-the-art LensLib pipelines like OmniLens (25.14 dB) and ZEBASELib-PT (23.83 dB), as well as general-purpose restoration models like S3Diff (24.57 dB).
- Inference Speed: Operating RSTB blocks at $1/8$ feature resolution allows FoundCAC to run in 0.417 seconds on a $1920 \times 1280$ image on an RTX 4090—almost twice as fast as SwinIR ($0.782$s) and orders of magnitude faster than diffusion-based methods ($>9$s).
- Real-World Photos (RealLens-Snap): Effectively sharpens extreme blur on single lenses, suppresses severe purple fringing, recovers high-frequency structures on out-of-domain metalens imagery, and refines high-end DSLR captures without over-sharpening or artifact induction.
Limitations and Future Work
- Coupled Real-World Effects: The current pipeline focuses primarily on optical aberration blur. It does not explicitly account for depth-of-field defocus, under-display diffraction, low-light sensor read noise, or complex ISP pipelines simultaneously.
- Optical System Types: While handling refractive and simple metalens configurations, extension to diffractive optical elements (DOEs) and metasurfaces within the EAOD loop remains future work.
- Downstream Adaptability: The authors note the need for lightweight fine-tuning recipes when adapting FoundCAC to specialized edge sensors.
Summary updated