Image reconstruction tasks—such as super-resolution, denoising, and depth map upsampling—are fundamentally ill-posed. Because a single degraded image can correspond to an infinite number of high-resolution or clean possibilities, modern computer vision algorithms frequently rely on a high-quality "guidance" or "reference" image. The core intuition is simple: borrow sharp edges and intricate textures from a clear reference source to fill in the blanks of a degraded target image.
However, this process introduces a critical vulnerability known as texture leakage or texture copying. If the reference image contains structural patterns, stylistic elements, or noise that do not exist in the target image, naive transfer mechanisms will blindly stamp these irrelevant details onto the final output. This results in severe visual artifacts, ghost structures, and what Yulun Zhang et al. describe in Texture Hallucination for Large-Factor Painting Super-Resolution as "identity loss"—where the reference's characteristics inadvertently overwrite the true content and color profile of the target. As observed by Yijun Li et al. in Joint Image Filtering with Deep Convolutional Networks, managing these structural inconsistencies is the defining challenge of reference-guided imaging.
To solve this, a major research direction has emerged focused on selective texture transfer: designing mechanisms that can distinguish between meaningful shared structures and misleading, irrelevant textures.
The Geometric and Domain Perspective Gaps
Why do standard matching algorithms fail and cause texture leakage? The root cause lies in the physical and environmental discrepancies between the target and reference captures, which can be categorized into geometric and domain gaps.
First, there is a geometric perspective gap. Reference images are rarely perfectly aligned with the target; they are often captured from different camera angles, scales, or rotations. Kelvin C.K. Chan et al. formalize this challenge in Robust Reference-Based Super-Resolution via C2-Matching as the "transformation gap." When spatial misalignments occur, conventional patch-matching techniques map textures to the wrong coordinates, creating corrupted outputs. In multi-view setups, Ri Cheng et al. demonstrate in Geometry-Aware Reference Synthesis for Multi-View Image Super-Resolution that viewpoint and scale variations prevent direct texture reuse. This spatial misalignment is particularly acute in satellite imaging, where Jeongho Min et al. note in Bridging the Domain Gap: A Simple Domain Matching Method for Reference-based Image Super-Resolution in Remote Sensing that large-scale perspective shifts completely disrupt standard texture transfer even when visual distribution gaps like lighting are bridged.
Second, there is a domain and content gap. When performing multimodal filtering (e.g., using an RGB photo to guide a near-infrared or depth map), the reference image contains unique textures that lack any physical counterpart in the target modality. As Pingfan Song explains in Multimodal Image Denoising based on Coupled Dictionary Learning, failing to separate these modality-specific features causes the model to hallucinate non-existent structures. When multiple reference images are available, the problem compounds; Marco Pesavento et al. show in Attention-based Multi-Reference Learning for Image Super-Resolution that without an explicit filtering mechanism, models will force inappropriate textures from a diverse reference pool onto the target structure.
Isolating Mutual Structures
To prevent texture copying, early frameworks focused on mathematical optimization to isolate mutual structures—edges and boundaries that are explicitly present in both the target and reference images.
In Mutual-Structure for Joint Filtering, Xiaoyong Shen et al. pioneered a joint filtering framework that explicitly suppresses edges unique to the reference image. By calculating a mutual-structure weight, the filter ensures that only shared structural boundaries are enhanced, effectively eliminating ghost artifacts in depth restoration and stereo matching.
To make this boundary selection robust to noise, Bumsub Ham et al. introduced a framework in Robust image filtering using joint static and dynamic guidance that combines static guidance (from the reference) with dynamic guidance (from the evolving output itself). By optimizing a non-convex objective function, the model dynamically determines whether a reference edge is a meaningful boundary or a superficial texture, preventing leakage. Similarly, Wei Liu et al. developed a generalized framework in A Generalized Framework for Edge-Preserving and Structure-Preserving Image Smoothing using a truncated Huber penalty function to explicitly manage structural inconsistencies, ensuring that target boundaries are preserved without copying extraneous reference details.
Deformable Alignment and Attention Mechanisms
With the advent of deep learning, researchers shifted from rigid optimization to flexible network architectures capable of dynamically warping and filtering reference features. Standard convolutional layers operate on a fixed grid, making them highly sensitive to geometric misalignments.
To overcome this, Beomjun Kim et al. proposed Deformable Kernel Networks for Joint Image Filtering. Instead of regressing pixels through static grids, the network learns to adaptively select the optimal neighboring pixels and weights for every single coordinate. This spatial flexibility prevents misleading gradients in the guidance image from forcing incorrect texture replication.
In reference-based super-resolution (Ref-SR), attention mechanisms have been redesigned to handle these spatial transformations. Jiezhang Cao et al. introduced Reference-Based Image Super-Resolution with Deformable Attention Transformer (DATSR), which pairs the global modeling power of Transformers with deformable attention. This allows the network to precisely align and warp high-frequency textures across significant scale and color variations, filtering out irrelevant reference information before it can generate artifacts. This builds upon texture transformer concepts, such as those explored by Changhong Liu et al. in A Novel Deep-Learning-Based Enhanced Texture Transformer Network for Reference Image Super-Resolution, which aim to optimize channel learning for high-fidelity texture integration.
Task Decoupling and Feature Insulation
A final core intuition in modern selective transfer is architectural isolation: separating the reconstruction of the target's global layout from the synthesis of fine textures. When a network tries to perform both tasks simultaneously, adversarial and perceptual losses often cause content-texture interference, leading the model to force mismatched details onto the output.
To break this interference, Yixuan Huang et al. proposed the Task Decoupled Framework for Reference-based Super-Resolution. Their framework explicitly decouples the pipeline into two independent stages: the model first upscales the structural content of the low-resolution input, and only then "paints" the high-frequency textures from the reference image. If the reference image is found to have low similarity, the painting stage can be selectively suppressed, preventing the misuse of irrelevant data.
Similarly, Xiaoyong Mei et al. introduced a two-stage feature reuse framework in A feature reuse framework with texture-adaptive aggregation for reference-based super-resolution. By first training the network strictly with reconstruction loss to lock down the target's true structural features, and subsequently reusing those features during texture aggregation, they insulate the output from the distorting effects of complex reference textures. This ensures that even when a reference image is highly irrelevant, the model maintains structural fidelity without introducing reference-induced artifacts.
Key Foundational Work
Multimodal Image Denoising based on Coupled Dictionary Learning (2018) — contributed 1 concept elements to this direction.
Mutual-Structure for Joint Filtering (2015) — contributed 1 concept elements to this direction.
Robust image filtering using joint static and dynamic guidance (2015) — contributed 1 concept elements to this direction.