When we think about image-to-image translation—such as adding makeup to a face, removing motion blur, or converting a non-contrast CT scan into a contrast-enhanced one—the naive approach is to feed a source image into a generator network and ask it to synthesize the target image from scratch. However, this forces the network to relearn complex spatial structures, high-frequency background textures, and global lighting conditions that should remain completely unchanged.
A powerful paradigm shift across computer vision literature bypasses this redundancy through additive residual mapping. Instead of modeling the entire target distribution, the generator is constrained to predict only a delta map ($\Delta$) representing the precise pixel-level or feature-level differences between the source and target states. The final output is simply computed as:
$$\text{Target} = \text{Source} + \Delta$$
The Default State: Identity Preservation
Why does this simple architectural constraint work so well? The primary intuition lies in how neural networks handle initialization and optimization. By structuring the network to output an additive delta, the default mathematical operation becomes an identity function ($\Delta = 0$). If a region of the image requires no modification, the network merely needs to output zeros.
This property is crucial for preserving a subject's identity and background integrity. For example, in facial attribute manipulation, traditional generative adversarial networks (GANs) often introduce global artifacts or alter the background when trying to change a single feature like adding glasses. As demonstrated in Learning Residual Images for Face Attribute Manipulation, applying an $L_1$ regularization penalty to a sparse residual image forces the model to modify only the pixels strictly necessary for the transformation, leaving the rest of the scene untouched.
Similarly, Chang et al. utilize this approach in asymmetric style transfer to apply and remove makeup without altering the underlying skin tone or background lighting. In real-world scenarios where backgrounds are messy or complex, Zheng shows that generating an additive residual makeup layer successfully decouples style modifications from identity content, preventing the distortion of non-face regions.
Simplifying the Optimization Landscape
Beyond preserving unchanged regions, residual mapping fundamentally alters what the network has to learn. In tasks like image deblurring or super-resolution, the network does not need to understand how to render a tree, a car, or a building; it only needs to isolate and reverse the degradation pattern.
In DeblurGAN, Kupyn et al. leverage a residual mapping generator to focus exclusively on predicting the motion blur pattern. Because the model is grounded by the structural template of the blurred input, the optimization process becomes significantly faster and more stable. This grounding effect is also evident in high dynamic range (HDR) synthesis; A et al. demonstrate that predicting a residual difference between a low-quality input and an HDR target yields highly stable training and prevents structural drift.
When applied to face aging, Hu et al. show that learning a "residual face" between age states isolates age-specific textures (like wrinkles) from age-irrelevant features (like hair, clothing, and environment). This decoupling ensures that the core structural features of the individual remain perfectly intact across decades of simulated aging.
Generalizing to Latent and Feature Spaces
The intuition of additive residual mapping is not restricted to raw pixel values; it applies equally to intermediate feature representations and latent spaces.
For instance, in structured domain adaptation, Hong et al. train a generator to produce a feature-level delta map that is added to synthetic features, aligning them with real-world feature distributions without disrupting the underlying semantic layout. In text-driven image editing, Dong et al. introduce residual feature transformation units that treat text prompts as additive modifications to existing image features, ensuring that changes occur only where dictated by the text.
This concept extends all the way to the latent codes of pre-trained generative models. In StyleGAN inversion and attribute editing, Patashnik et al. show that instead of re-encoding an entire image into a new latent vector, the model can learn a residual vector that shifts a fixed, identity-preserving latent code toward a target state. Similarly, in the text domain, Mai et al. utilize a residual architecture to learn small offsets within an embedding space, maximizing content preservation during style transfer.
High-Fidelity Domain Shifts
The power of treating the source image as a fixed structural template becomes highly apparent in specialized domains like medical imaging and scene relighting, where structural errors can have catastrophic consequences.
In medical image translation, La Barbera et al. apply this framework to translate non-contrast CT scans into contrast-enhanced ones. By forcing the network to learn only the additive or subtractive signal representing the contrast agent's physiological distribution, the underlying anatomy remains perfectly registered. Wei et al. mirror this biological mechanism by predicting the localized residual differences for multi-phase liver CT generation. For anomaly detection, the unsupervised framework Brainomaly "heals" brain MRIs by generating a localized additive map to remove structural signs of disease while strictly preserving the patient's unique brain anatomy.
In lighting and shadow manipulation, El Helou et al. note that predicting a residual difference map preserves high-frequency textures during illumination transfer. This allows models like ARShadowGAN, developed by Liu et al., to generate realistic shadows for virtual objects by predicting a residual shadow map that applies precise intensity reductions directly onto the original background texture.
Key Foundational Work
ELEGANT: Exchanging Latent Encodings with GAN for Transferring Multiple Face Attributes (2018) — contributed 2 concept elements to this direction.
Global and Local Consistent Age Generative Adversarial Networks (2018) — contributed 2 concept elements to this direction.
PairedCycleGAN: Asymmetric Style Transfer for Applying and Removing Makeup (2018) — contributed 2 concept elements to this direction.