Controllable and Interactive Image Restoration

56 concepts46 papers1 proposal

Image restoration is fundamentally ill-posed. For any degraded image—whether it is blurry, noisy, or compressed—there exist infinitely many clean images that could have generated it. Traditional deep learning approaches treat this as a deterministic mapping problem, training a "black-box" network to output a single, statistically optimal reconstruction.

However, what is mathematically optimal (such as minimizing Mean Squared Error) often fails to align with human preference, resulting in over-smoothed textures or artificial hallucinations. Furthermore, the ideal balance between noise reduction, detail preservation, and exposure is highly subjective and context-dependent.

To bridge this gap, a prominent research direction focuses on controllable and interactive image restoration. Instead of delivering a static, all-or-nothing output, these frameworks expose internal degradation and quality parameters to the user at inference time. By allowing humans to dynamically steer the restoration process via sliders, text prompts, or sparse annotations, these systems transform image restoration from a rigid translation task into an interactive, co-creative process.

Decoupling Latent Spaces for Real-Time Tuning

To give users continuous control over a deep neural network's output without requiring costly retraining or multiple forward passes, models must decouple the image's structural content from its degradation parameters within the network's latent space.

In CFSNet: Toward a Controllable Feature Space for Image Restoration, Wang et al. demonstrated that by adaptively coupling features from specialized network branches, a single control variable can smoothly interpolate between different reconstruction objectives, such as smoothness versus sharpness. This architecture allows users to navigate the feature space dynamically.

Similarly, Jiang et al. applied this principle to compression artifacts in Towards Flexible Blind JPEG Artifacts Removal by decoupling latent quality factors from underlying image features. By exposing this latent quality factor as a continuous parameter, a single model provides a continuous range of restoration strengths, allowing users to interactively balance artifact suppression against the preservation of fine textures based on subjective preference.

[Degraded Input] ---> [Feature Extraction] ---> [Decoupled Latent Space]
                                                       |
                                        [User Control Parameter / Slider]
                                                       |
                                                       v
[Restored Output] <--- [Reconstruction] <--- [Feature Modulation Module]

To make this interactive tuning practical for real-time applications like video processing, the computational overhead of adjusting these parameters must be minimized. Zhang et al. addressed this challenge in Real-Time Controllable Denoising for Image and Video by introducing a lightweight "Noise Decorrelation" module at the final layer of a standard architecture. This allows users to adjust denoising intensity via a slider with only a single network inference pass, achieving a 2000x speedup in editing latency without sacrificing restoration quality.

A recurring challenge in generative image restoration is balancing fidelity (how closely the output matches the ground-truth structure) and perception (how realistic and visually pleasing the output looks).

For instance, in face restoration, a model that relies too heavily on generative priors might create a perfectly sharp face that no longer looks like the original person. To resolve this, Wang et al. proposed Panini-Net: GAN Prior Based Degradation-Aware Feature Interpolation for Face Restoration, which uses a mask-based interpolation module to let users manually navigate the trade-off between identity consistency and generative enhancement. In a similar vein, Liu et al. developed Latent Harmony: Synergistic Unified UHD Image Restoration via Latent Space Regularization and Controllable Refinement, utilizing low-rank adaptation to effectively decouple fidelity and perception in ultra-high-definition image restoration.

This trade-off is also highly apparent in hardware-specific restoration. In Controllable Image Restoration for Under-Display Camera in Smartphones, Kwon et al. introduced a noise-adaptive network that allows users to tune the balance between sharpness and noise reduction. This recognizes that a portrait photographer might want to preserve soft skin textures at the expense of some noise, while a landscape photographer might prioritize crisp edges.

Language-Guided and Prompt-Aware Granularity

While global sliders are effective for uniform degradations like sensor noise or JPEG compression, real-world degradation is often composite and localized. A user might want to remove a harsh shadow or a window reflection while preserving atmospheric haze or rain for artistic effect. To provide this granular control, modern frameworks leverage natural language prompts and sparse visual annotations.

In Prompt-Aware Controllable Shadow Removal, Chen et al. introduced PACSRNet, which allows users to target specific shadows using simple prompts like a single dot, a rough line, or a subject mask. This prevents the all-or-nothing removal typical of automated systems and maintains realistic lighting in complex scenes.

For multi-degradation scenarios, Guan et al. proposed Referring Flexible Image Restoration, a framework where natural language instructions direct the model to selectively eliminate certain artifacts (e.g., motion blur) while intentionally leaving others (e.g., snow) intact.

This concept is extended to scientific and multi-modal domains by Kurinchi-Vendhan et al. in Seeing Through the PRISM: Compound & Controllable Restoration of Scientific Images. Here, a compositional latent space enables experts to specify exactly which distortions to filter out via natural language, preventing black-box models from inadvertently erasing critical scientific data during automated preprocessing.

Continuous Dynamics and Human-in-the-Loop Workflows

Beyond single-step parameter adjustments, some researchers frame controllable restoration as a continuous or sequential trajectory. By treating the enhancement process as a dynamic system, users can sample intermediate states along a transformation path to find their preferred output.

In Continuous Exposure Learning for Low-light Image Enhancement using Neural ODEs, Donggoo Jung reformulated curve adjustment as a Neural Ordinary Differential Equation (NODE). By treating the brightening process as a continuous flow, it gives users intuitive, continuous control over exposure levels during inference.

Similarly, Lanqing Guo developed ReLLIE: Deep Reinforcement Learning for Customized Low-Light Image Enhancement, a deep reinforcement learning framework that treats low-light enhancement as a sequential process. The model learns a policy to estimate pixel-wise enhancement curves, allowing users to choose their preferred level of illumination from the agent's step-by-step actions.

[Low-Light Input] ---> [NODE / DRL Agent] ---> Step 1 (Low Exposure)
                                         ---> Step 2 (Medium Exposure)  <-- User Choice
                                         ---> Step 3 (High Exposure)

When dealing with severe physical damage, such as in historical photo restoration, fully automated artifact detection often fails. Seung-Yeon Back demonstrated in Preserving Old Memories in Vivid Detail: Human-Interactive Photo Restoration Framework that incorporating human-guided masking for inpainting and stage-by-stage parameter tuning dramatically improves user satisfaction and identity preservation compared to autonomous baselines. By keeping the human in the loop, these frameworks successfully navigate the deeply subjective boundaries of visual restoration.

Key Foundational Work

Guided color consistency optimization for image mosaicking (2017) — contributed 1 concept elements to this direction.
Reflection Separation Using Guided Annotation (2017) — contributed 1 concept elements to this direction.
Improved temporal psychovisual modulation for backward-compatible stereoscopic display (2014) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_controllable_image_restoration_a3ae1823,
  title = {controllable image restoration},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/controllable-image-restoration/rdr_a3ae182382b14065a16cc7841cd778fb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_controllable_image_restoration_a3ae1823_v1,
  title = {controllable image restoration},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/controllable-image-restoration/rdr_a3ae182382b14065a16cc7841cd778fb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake