Mining the Edit Moment: Perceptually Scheduled, Spectrally Anchored Triplets for Composed Image Retrieval

Composed image retrieval lets a user search with a picture plus an edit instruction, for example a photo of a sneaker plus "make it black and add spikes." Training such a system needs triplets: a reference image, the instruction, and the correct target image. Collecting these by hand is slow, so the human datasets stay near 30,000 examples. The usual workaround is to generate targets with a diffusion model, which is a generator that turns random noise into images by gradually removing noise. That workaround has a quiet flaw. A text-guided edit does not change the image smoothly as edit strength grows. The image barely moves, then flips over a narrow range, then stalls. A fixed edit strength therefore produces some targets where the edit never happened and others where the whole image drifted, including lighting, color, and background that the instruction never mentioned. Either way the triplet is mislabeled.

We propose to build triplets from real reference images and to treat each edit as a trajectory, not a single sample. We run the edit at many strengths, measure how fast the image changes perceptually, and pick the target just after the burst of change, when the described edit is complete. While generating, we hold the reference's low-frequency Fourier amplitude fixed, which roughly encodes global color and illumination, so that undescribed appearance drift is suppressed. The same trajectory also supplies something the field lacks: graded hard negatives, meaning frames from before the burst that look almost like the reference but do not yet satisfy the instruction. The claim is that triplet quality, not triplet count, is the bottleneck, and that the perceptual change curve of an edit is a cheap, label-free signal for that quality.

Core Claim

A synthetic triplet is correct only if the reference-to-target change is complete and confined to what the text describes. Existing synthesis pipelines control neither property. CompoDiff builds SynthTriplets18M with a language model writing instructions and Stable Diffusion with Prompt-to-Prompt rendering both images at a fixed procedure. Its 15.33 mAP@5 on CIRCO is beaten by the language-only LinCIR at 19.71. This motivates our reading: 18.8 million image triplets lose to zero image triplets, which suggests noisy targets rather than too little data.

The morphing literature supplies the measurement tool. IMPUS uses a binary search on LPIPS, a learned perceptual distance between images, to space frames evenly in perceived change. DiffMorpher redistributes interpolation steps along the cumulative perceptual-distance curve and reports that change concentrates in burst zones. AID replaces search with a fixed Beta-distribution prior centered on the midpoint. All three use the curve to make animations look smooth. We use the location of the burst as a label: it marks where the instruction becomes true. Retrieval needs one well-chosen point and its neighbors, not an evenly spaced sequence.

The appearance control comes from two lines of work. Latent AdaIN matches the channel mean and variance of diffusion latents to a reference. It is used in DiffMorpher, Style Injection in Diffusion, and TALE. Because it acts globally per channel, it also rescales the edge and texture energy that carries the edit. Amplitude-phase separation comes from FDA, which swaps only a small low-frequency amplitude window between images while keeping phase, and from Phase Consistent Ecological DA, which shows that phase carries object identity. Both use this separation on pixels for segmentation adaptation, not on diffusion latents. We restrict normalization to the low-frequency amplitude band of the latent. That anchors global appearance to the reference and leaves the phase and high frequencies free for the edit.

Research Program

Trajectory generation. Reference images come from CC3M, which avoids the COCO images behind CIRCO. A language model writes an instruction and an edited caption from each reference caption. We invert the reference into the latent space of an off-the-shelf diffusion model. We then define an edit path parameterized by strength $s \in [0,1]$, using interpolated attention from reference caption to edited caption in the style of AID. We decode $K$ frames (about 16) at a coarse grid of strengths and refine near the steepest segment with a few bisection steps, as in IMPUS. This is offline data construction, so the cost of repeated sampling, which is the main drawback of reactive schedules, is paid once per triplet rather than per query.

Choosing the target. Let $d(s)$ be the LPIPS distance between the frame at strength $s$ and the reference, and let $v(s) = d'(s)$ be the perceptual velocity. We take the target strength to be the first strength after the velocity peak where the velocity has fallen back below a fraction $\gamma$ of that peak:

$$s^\ast = \min\{\, s > \arg\max_u v(u) \;:\; v(s) < \gamma \max_u v(u) \,\}.$$

At that point the burst is over and further strength mostly adds drift. If no clear peak exists, meaning the velocity is roughly flat, the trajectory is discarded as an edit the generator cannot cleanly perform. Frames at strengths below the burst onset become hard negatives for that query, ordered by $d(s)$.

Spectral anchoring. At each early-to-middle denoising step, let $z$ be the edited latent and $z_r$ the reference's latent at the same step. Write $\mathcal{F}$ for the 2D Fourier transform applied per channel, with amplitude $A$ and phase $\Phi$, and let $M_\beta$ be a centered low-frequency window of relative radius $\beta$. We replace

$$z \leftarrow \mathcal{F}^{-1}\Big( \big[ M_\beta A(z_r) + (1-M_\beta) A(z) \big]\, e^{i\Phi(z)} \Big).$$

This keeps the edited latent's phase and high-frequency amplitude while borrowing the reference's coarse appearance. When the language model tags an instruction as photometric, such as a color, lighting, or time-of-day change, anchoring is switched off, because the edit then lives in exactly that band. The window size $\beta$ is tuned once on a small held-out set. FDA reports that this window is sensitive, so we sweep it rather than assume it.

Retrieval training. We hold the retrieval model fixed so that differences come from data alone. We use CLIP ViT-L with the light LoRA image-encoder tuning and Slerp fusion of Slerp + TAT, trained with a contrastive loss over targets. Each batch includes the trajectory hard negatives.

Baselines

The naive reference point is the same retrieval model trained on our pipeline with a fixed edit strength ($s=0.5$, and separately $s=0.8$) and no anchoring. It isolates whether the schedule matters at all. The fixed-prior ablation samples $s$ from AID's Beta prior instead of locating the burst per trajectory. If it matches us, per-trajectory measurement is unnecessary. The normalization ablation swaps spectral anchoring for DiffMorpher-style global latent AdaIN, and separately for no normalization. It isolates the spectral restriction. A hard-negative ablation drops the pre-burst frames.

The data competitor is an equal-size, randomly drawn subset of SynthTriplets18M from CompoDiff, and the full set where compute allows, trained in the identical model with identical steps and tuning budget. It is the strongest existing source of image triplets.

The strongest existing methods, reported from their papers, are LinCIR and Slerp + TAT. LinCIR reports 19.71 mAP@5 on CIRCO and 45.11 average R@50 on FashionIQ with ViT-G. Slerp + TAT reports 33.98 R@1 on CIRR and 18.46 mAP@5 on CIRCO. We will also rerun both at ViT-L so that backbone size does not decide the comparison. MulVec, a training-free method, reports relative CIRCO gains of 19.7 to 23.0 percent over training-free methods, but no reported absolute result was found on Lacuna for MulVec. We will rerun it if code is available.

Beating the field means two things. First, at matched backbone, our data must exceed Slerp + TAT's own training recipe on CIRR R@1 and CIRCO mAP@5. Second, at matched triplet count, our data must beat SynthTriplets18M. We also report whether adding our data to LinCIR's language-only training improves on its 19.71.

Why not just…?

Why not just filter SynthTriplets18M with a CLIP similarity score? Filtering is the most serious threat, because it is cheap and attacks the same label noise. A text-image score rewards the target matching the edited caption. It does not penalize undescribed drift, and it cannot tell an incomplete edit from a complete one when both partly match. We will test this directly. We filter SynthTriplets18M, and our own fixed-strength triplets, to the same count using CLIP directional agreement. If filtered fixed-strength triplets match scheduled ones, we will drop the schedule and report spectral anchoring plus negatives as the contribution.

Why not just skip image triplets, as LinCIR does? LinCIR's lead over CompoDiff is the motivating evidence for this proposal, so the alternative is real. Language-only training, however, never sees how an image changes. The CIR failure modes the survey lists, such as loss of fine detail and confusion with the reference, are visual. Trajectory negatives target the reference-confusion failure. We will measure how often each model ranks the reference itself, or a pre-burst frame, above the true target on CIRR. Combining our data with LinCIR's training tests whether the signals are complementary rather than competing.

Why not just use the Beta prior from AID instead of measuring each trajectory? AID's prior assumes the burst sits near the midpoint. Edits differ widely: adding a small object differs from changing a scene's season. The fixed-prior ablation settles this. We also report the empirical distribution of $s^\ast$. If it is tightly centered, the prior suffices and the measurement cost is wasted.

Why not just use plain latent AdaIN? Global AdaIN rescales all spatial frequencies of each channel, including the edge energy that encodes added or removed objects. The FDA results show that confining appearance transfer to low-frequency amplitude preserves boundaries. The normalization ablation measures this on triplets through an undescribed-change score, described below, and on downstream retrieval.

Proposed Experiments

The first check validates the triplets themselves before any retrieval training. On 2,000 sampled triplets per pipeline, human raters judge whether the edit is complete and whether anything undescribed changed. An automatic proxy computes LPIPS between reference and target outside a region where the edit is localized, using a segmentation of the changed area. The hypothesis is that burst-selected, anchored triplets show higher completeness and lower undescribed change than both fixed-strength and SynthTriplets18M triplets.

As a sanity check on the velocity measurement, we run our curve estimator on MorphBench pairs and confirm that it finds bursts where DiffMorpher reports concentrated change. That paper reports a perceptual path length (PPL) of 18.19 versus 27.80 for DDIM in metamorphosis.

The main experiment trains the fixed retrieval model on 1M triplets from each pipeline and evaluates zero-shot on CIRR (R@1, R@5, R@50), CIRCO (mAP@5, mAP@50), FashionIQ (average R@10 and R@50), and GeneCIS.

Two results would be decisive. The first is a scaling comparison: our 1M against SynthTriplets18M subsets of 1M, 4M, and 18M. If our 1M matches the full 18M, quality has beaten quantity. The second is a breakdown by instruction type: photometric, object addition or removal, and attribute changes. The spectral anchoring gain should appear for non-photometric edits and vanish, by construction, for photometric ones. If the gain instead appears uniformly, anchoring is acting as generic regularization rather than the mechanism we claim, and we will say so.