In image processing, we often face "inverse problems": we have a degraded observation—blurry, noisy, or undersampled—and we want to recover the original, clean image. Mathematically, this is usually framed as minimizing an energy function with two parts: a data fidelity term (making sure our solution matches the observed data) and a regularization term (making sure the solution looks like a real image).
The challenge is that these two terms often "speak different languages." The data term might involve complex physics or sensor models, while the regularization term might be non-differentiable, like the $L^1$ norm or Total Variation (TV). Solving them together in one giant optimization step is computationally grueling.
Variable splitting is the strategy of "dividing and conquering" this complexity. Instead of solving one hard problem, we break it into a sequence of simpler, iterative sub-problems.
The Intuition of Decoupling
The core idea behind variable splitting is to introduce an auxiliary variable that "splits" the objective function. If we want to minimize $f(x) + g(x)$, we instead minimize $f(x) + g(z)$ subject to the constraint that $x = z$.
This might seem like making the problem more complicated, but it allows us to decouple the physics of the imaging system from the statistics of the image itself. As Kai Zhang and Lei Zhang demonstrate in Learning Deep CNN Denoiser Prior for Image Restoration, this modularity is incredibly powerful. It allows a single algorithm to solve multiple tasks—like deblurring, super-resolution, and inpainting—simply by swapping out the data fidelity term while keeping the same "denoising" prior.
The Mathematical Engines: HQS and ADMM
Two primary algorithms drive this research area: Half-Quadratic Splitting (HQS) and the Alternating Direction Method of Multipliers (ADMM).
In HQS, the constraint $x=z$ is relaxed into a penalty term. This results in two alternating steps:
- The Proximal Step: Solving for $x$ given $z$. This usually handles the data fidelity.
- The Denoising Step: Solving for $z$ given $x$. This handles the regularization.
Majed El Helou and Sabine Süsstrunk show in Fourier-Domain Optimization for Image Processing that if the data term involves convolutions (like blur), the first step can be solved almost instantly in the Fourier domain. Meanwhile, the second step often reduces to simple "soft-thresholding" or lookup tables, as noted by Tushar Sandhan in his work on removing particulate matter and eyeglass reflections.
ADMM, used by Wotao Yin and Yiqiu Dong for Cauchy noise removal, adds a "dual variable" to the mix. This ensures that the algorithm eventually satisfies the $x=z$ constraint exactly, providing stronger convergence guarantees for difficult, non-convex problems.
Speed through Domain Transformation
One of the biggest reasons variable splitting works so well in practice is its compatibility with fast transforms. When we decouple the variables, the sub-problems often take forms that are computationally "cheap."
For example, Mariano Tepper and Guillermo Sapiro show in L1 Splines for Robust, Simple, and Fast Smoothing of Grid Data that by using Split-Bregman (a close relative of ADMM), they can use the Discrete Cosine Transform (DCT) to solve the quadratic sub-problem. Similarly, Serge L. Shishkin and Gregory S. Hagen leverage separable sensing operators to replace slow iterative steps with highly efficient Sylvester equation solvers, speeding up reconstruction by orders of magnitude.
Hybrid Intelligence: Plug-and-Play and Deep Unfolding
The most recent evolution of this field involves merging these classical optimization frameworks with deep learning.
In the Plug-and-Play (PnP) framework, researchers realized that the "regularization" sub-problem in HQS or ADMM is essentially a denoising task. Instead of using a mathematical formula like Total Variation, they "plug in" a pre-trained deep neural network as the denoiser. Kai Zhang demonstrated that this allows a single CNN to act as a universal prior for various restoration tasks.
Taking this a step further, Deep Unfolding (or Unrolling) treats the iterations of the splitting algorithm as layers in a neural network. Mouna Gharbi and Jean-Christophe Pesquet explore this in Analyse comparative d'algorithmes de restauration en architecture dépliée, showing that an unfolded semi-quadratic (U-HQ) architecture can outperform standard black-box networks by baking the physics of the problem directly into the network's structure.
Multi-Tasking through Splitting
Because variable splitting can handle multiple constraints simultaneously, it is ideal for "entangled" problems. Angelica I Aviles-Rivero and colleagues have applied this to medical imaging, where they jointly reconstruct and register MRI images. By splitting the problem, they can solve for image intensity and motion vectors in a coordinated sequence, leading to sharper results than solving either problem in isolation.
Whether it's removing clouds from satellite images or reconstructing images from event cameras, the core philosophy remains the same: break the complex whole into manageable, specialized parts.