Non-Commutative Image Restoration Pipelines

This direction is merged into another direction: compound image degradation, restoration sequence optimization. Its last version stays readable and citable here.

Version 1 of 128 papers43 conceptsfrozen 2026-10-04

In classical image processing, we often treat restoration as a checklist: remove the noise, fix the blur, and then upscale. However, recent research reveals that image restoration is fundamentally non-commutative. The order in which you apply these operations—the "restoration sequence"—drastically changes the final output quality. Applying a tool at the wrong time doesn't just lead to sub-optimal results; it can irreversibly "bake" artifacts into the image, making further restoration impossible.

The Problem of Operator Order

The core challenge in complex image restoration is that early processing steps alter the statistical characteristics of the remaining degradations. As Hui Xiong notes in Digital Image Processing using the Fast Fourier Transform, applying sharpening before noise reduction can amplify artifacts to a level where they become indistinguishable from the actual signal.

This non-commutative nature is most visible in the "Demosaicking vs. Denoising" dilemma. Qiyu Jin and Jean-michel Morel demonstrate in A Review of an Old Dilemma that while denoising first preserves noise statistics, it risks losing spatial resolution. Conversely, demosaicking first transforms simple white noise into complex, spatially correlated noise that standard algorithms struggle to handle.

Why Fixed Pipelines Fail

Most traditional models use a fixed, linear sequence of operations. However, Mehran Jeelani and colleagues argue in Expanding Synthetic Real-World Degradations that these static pipelines fail in the wild because real-world noise and blur don't follow a predictable order. A model trained only to "denoise then deblur" may fail to generalize when it encounters a "blurred noise" artifact.

Similarly, in inclement weather restoration, Wenhan Yang and Robby T. Tan show in their work on JORDER that heavy rain involves both discrete streaks and mist-like accumulation. Treating these as independent problems with a fixed sequence often leads to persistent artifacts, as the first step may destroy information the second step needs.

Agentic Orchestration and Dynamic Planning

To solve the ordering problem, a new generation of "agentic" systems has emerged. Instead of a single model, these systems use Large Language Models (LLMs) or Vision-Language Models (VLMs) to act as managers that perceive the image and plan a custom restoration sequence.

  • RestoreAgent: Haoyu Chen and Wenbo Li introduced RestoreAgent, which models restoration as a sequential decision-making process. It understands that an incorrect order can prevent subsequent models from functioning.
  • AgenticIR: Kaiwen Zhu and Jinjin Gu developed AgenticIR, which explicitly schedules steps—like removing noise before super-resolution—to prevent upscaling artifacts into unrecognizable patterns.
  • JarvisIR: In the context of autonomous driving, Yunlong Lin and Zixu Lin demonstrate with JarvisIR that automated reasoning can identify restoration sequences that are often counter-intuitive to human experts but yield significantly better perception metrics.

Building Intuition: The Super-Resolution Trap

The most common example of non-commutativity is the interaction between artifact removal and super-resolution. If you perform super-resolution (upscaling) on an image with rain streaks or noise, the model treats those artifacts as "details" to be enhanced. This "bakes" the noise into the high-frequency structure of the image. Once upscaled, a specialized de-noiser can no longer recognize the noise because its shape has been transformed, rendering the restoration impossible. By prioritizing artifact removal before upscaling, the pipeline ensures that the upscaler only enhances clean, structural information.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_non_commutative_image_restoration_pipeli_14e87d03,
  title = {Non-Commutative Image Restoration Pipelines},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/non-commutative-image-restoration-pipelines/rdr_14e87d030fab4754933eaba9ec8d8796},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_non_commutative_image_restoration_pipeli_14e87d03_v1,
  title = {Non-Commutative Image Restoration Pipelines},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/non-commutative-image-restoration-pipelines/rdr_14e87d030fab4754933eaba9ec8d8796/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}