LoRA of Change: Learning to Generate LoRA for the Editing Instruction from A Single Before-After Image Pair

Xue Song, Jiequan Cui, Hanwang Zhang, Jiaxin Shi, Jingjing Chen, Chi Zhang, Yu-Gang Jiang

Abstract

In this paper, we propose the LoRA of Change (LoC) framework for image editing with visual instructions, i.e., before-after image pairs. Compared to the ambiguities, insufficient specificity, and diverse interpretations of natural language, visual instructions can accurately reflect users' intent. Building on the success of LoRA in text-based image editing and generation, we dynamically learn an instruction-specific LoRA to encode the "change" in a before-after image pair, enhancing the interpretability and reusability of our model. Furthermore, generalizable models for image editing with visual instructions typically require quad data, i.e., a before-after image pair, along with query and target images. Due to the scarcity of such quad data, existing models are limited to a narrow range of visual instructions. To overcome this limitation, we introduce the LoRA Reverse optimization technique, enabling large-scale training with paired data alone. Extensive qualitative and quantitative experiments demonstrate that our model produces high-quality images that align with user intent and support a broad spectrum of real-world visual instructions.

Introduction: The Precision of Visual Instructions

While text-to-image editing has seen significant progress, natural language often suffers from ambiguity and a lack of specificity. A prompt like "make this look more artistic" can be interpreted in countless ways, leading to results that may not align with a user's specific intent. Visual instructions—providing a "before" and "after" example—are far more precise.

However, training models to follow these visual cues is challenging. Existing methods typically require "quad data"—two pairs of images showing the same transformation on different subjects—to learn generalizable edits. Such data is extremely scarce. Furthermore, models often suffer from "appearance leakage," where the model inadvertently copies the identity or specific pixels of the example images onto the target image, rather than just the abstract transformation.

The LoC Framework: Edits as Dynamic Weights

The LoRA of Change (LoC) framework addresses these challenges by treating an edit not as a pixel-manipulation task, but as a dynamic weight update.

The LoC framework applies a transformation learned from a visual instruction pair (A, A') to a new query image (B) to produce the target (B').

At the heart of LoC is a Hypernetwork consisting of a ViT encoder and a Transformer decoder. This network takes a before-after pair $<A, A'>$ as input and generates a set of Low-Rank Adaptation (LoRA) weights ($\Delta$). These generated weights are then injected into a frozen InstructPix2Pix model. Because LoRAs modify the model's internal behavior, they are uniquely suited to capturing the "delta" or the essence of the change between two states, such as a style transfer or an object addition.

LoRA Reverse: Symmetry as a Solution

To overcome the scarcity of quad data and prevent appearance leakage, the authors introduced LoRA Reverse optimization. This technique leverages the inherent symmetry of many edits. The model is trained such that if a generated LoRA $\Delta$ transforms image $A$ to $A'$, then the negative version of those weights ($-\Delta$) must be able to transform $A'$ back to $A$.

This symmetry constraint forces the hypernetwork to focus on the transformation itself rather than the specific content of the images. By using this trick, the model can be trained effectively on large-scale paired data (like the 3.7 million pairs in the SEED-Data-Edit dataset) without needing the elusive quad data.

Results and Generalization

LoC demonstrates remarkable zero-shot efficiency, generating weights in a single forward pass without the need for time-consuming test-time optimization. In quantitative evaluations on the InstructPix2Pix benchmark, LoC outperformed state-of-the-art methods like Analogist and VISII, achieving the lowest LPIPS (0.289) and the highest Visual CLIP score (0.214).

Qualitative results highlight the model's ability to generalize. A single visual instruction (e.g., turning a person into a caricature) can be successfully applied to diverse subjects, including animals and cartoon characters, while maintaining high fidelity to the original query image's identity. In user studies, LoC was preferred by participants 87.4% of the time over competing models.

Conclusion

By encoding editing instructions into reusable and interpretable LoRA weights, the LoRA of Change framework provides a robust and efficient solution for visual-instruction-based image editing. The introduction of LoRA Reverse training allows the model to scale using widely available paired data, paving the way for more precise and user-centric image manipulation tools.

Summary updated

Abstract

In this paper, we propose the LoRA of Change (LoC) framework for image editing with visual instructions, i.e., before-after image pairs. Compared to the ambiguities, insufficient specificity, and diverse interpretations of natural language, visual instructions can accurately reflect users' intent. Building on the success of LoRA in text-based image editing and generation, we dynamically learn an instruction-specific LoRA to encode the "change" in a before-after image pair, enhancing the interpretability and reusability of our model. Furthermore, generalizable models for image editing with visual instructions typically require quad data, i.e., a before-after image pair, along with query and target images. Due to the scarcity of such quad data, existing models are limited to a narrow range of visual instructions. To overcome this limitation, we introduce the LoRA Reverse optimization technique, enabling large-scale training with paired data alone. Extensive qualitative and quantitative experiments demonstrate that our model produces high-quality images that align with user intent and support a broad spectrum of real-world visual instructions.
BibTeX
@article{song2026lora,
  author = {Xue Song and Jiequan Cui and Hanwang Zhang and Jiaxin Shi and Jingjing Chen and Chi Zhang and Yu-Gang Jiang},
  title = {{LoRA of Change: Learning to Generate LoRA for the Editing Instruction from A Single Before-After Image Pair}},
  journal = {IEEE Trans. Image Process.},
  year = {2026},
  doi = {10.1109/tip.2026.3707785},
}
Figure