Paying for Synthetic Images Once: Amortized Latent Synthesis Across the Modality Gap

Several recent systems teach a model to "see" without paired image-text data by drawing synthetic pictures. To train an image captioner from text alone, they render images from captions with a text-to-image diffusion model (a generator that turns random noise into an image by repeated denoising). They then pass those images through a frozen vision encoder and train on the resulting features. The method works, but every training example costs a full diffusion run, and the only thing the downstream learner uses is a single embedding vector per image.

We propose to pay that cost only for a small seed set. On a few thousand generated images, we train a small diffusion model that lives in the encoder's embedding space and maps a text embedding to a plausible image embedding. We then use it to produce pseudo-image embeddings for millions of captions at almost no marginal cost. Because it reads and writes only encoder outputs, the synthesizer can be plugged into any frozen encoder and any downstream learner.

Core Claim

The gap between where a contrastive encoder puts text and where it puts images is the "modality gap": paired captions and images land in separate regions of the shared space instead of on top of each other. The idea succeeds only if that gap is more than a constant offset. Write a paired text and image embedding as

$$z_I = z_T + g + r(z_T) + \epsilon,$$

where $z_T$ and $z_I$ are the text and image embeddings, $g$ is the difference of the two modality means, $r(z_T)$ is the part of the remaining displacement that depends on the caption, and $\epsilon$ is caption-independent spread.

Cheap fixes model only $g$ and $\epsilon$. Centering methods such as I0T and GR-CLIP subtract modality means, and text-only noise-injection baselines add isotropic noise. Two findings suggest $r$ carries real structure. Mind the Gap traces the gap to narrow, differently oriented embedding cones. I0T finds the gap concentrated in a few dimension-specific activation peaks. Neither picture is a pure translation.

Our claim is that a learned embedding-space synthesizer, trained on roughly 5–10% of the diffusion-model calls that SynTIC spends, can recover most of SynTIC's benefit. It should also clearly beat mean-centering plus noise. If centering plus noise matches it, $r$ does not matter and the idea should be dropped. The experiments are built so this outcome is visible.

How this differs from existing work

Three groups of prior work bear on this proposal, in different roles.

SynTIC and Watching Synthetic Videos (WSV) motivate it. SynTIC renders Stable Diffusion images for text-only captioning and names generation cost as its main limitation. WSV synthesizes CogVideoX video latents and corrects them with a trained 3D-convolution "polisher." Both run the large generator for every training sample. WSV also couples its correction module tightly to a fine-tuned GPT-2. We instead run the large generator only on a seed set and amortize it.

ADAR supplies the method. It shows that a diffusion process acting directly on latent embeddings can produce useful training examples as a plug-in for many recommender backbones, but it works only on single-modality ID embeddings and never has to cross a modality gap.

I0T, GR-CLIP and Mind the Gap constrain the design. Centering is cheap and strong, so it is the baseline to beat. Mind the Gap also shows that zero gap is not always optimal, so the synthesizer exposes the target gap distance as an explicit setting rather than forcing it to zero.

Image-generation systems have used text-to-image-embedding priors as internal components. What we test is something different: whether such a prior, trained on a small budget of synthetic images, can replace per-sample pixel synthesis in alignment training. We also test whether it beats geometric centering.

Research Program

The pipeline has three stages, all built on a frozen CLIP encoder, with Long-CLIP as a second encoder to test portability.

Seed stage. We take $K$ captions from the training text corpus, render one Stable Diffusion image per caption as SynTIC does, and store only the image embeddings. $K$ ranges from 1k to 50k.

Synthesizer stage. We train a small conditional diffusion model, a few-layer MLP or transformer over the 512- or 768-dimensional embedding, to generate $z_I$ given $z_T$. Following SynTIC's own refinement step, the training objective also includes a contrastive term that keeps generated embeddings close to their caption and away from other captions. A guidance weight on the text condition, together with a mean-offset term, lets us set the centroid distance of the generated embeddings anywhere from I0T-style near-zero up to the native CLIP gap.

Student stage. We sample one or more pseudo-image embeddings for every caption in the corpus and train an unchanged downstream student on them. The primary student is SynTIC's captioning decoder. The secondary student is WSV's prompter and GPT-2, with the image encoder replaced by a frozen video-frame encoder and CogVideoX replaced by our synthesizer.

Diagnostics. Before any student is trained, we measure on real Flickr30k pairs how much of the residual $z_I - z_T - g$ is predictable from $z_T$. We fit both a linear regressor and the synthesizer itself and report explained variance. This number predicts in advance whether the method can beat centering. We also log total generator GPU-hours for every method, because neither SynTIC nor WSV reports one: no reported result found on Lacuna for SynTIC or WSV synthesis cost.

Baselines

Text-only naive reference. The captioner is trained on text embeddings, and at test time it is fed raw image embeddings with no correction. This measures the full damage of the modality gap. Within the file, the closest published reference is CapDec, which SynTIC reports at 26.4 BLEU-4 on MSCOCO.

Strongest geometric competitor. Centering in the style of I0T and GR-CLIP: during training, text embeddings are shifted by the gap vector $g$ estimated on a calibration set and perturbed with isotropic noise whose scale is tuned on validation. At test time, image embeddings are centered with I0T's training-free post-hoc correction. This is the most dangerous competitor because it needs zero generator calls. Our method must beat it by at least 1.5 BLEU-4 on MSCOCO to justify the seed cost.

Strongest synthetic-data method. Full SynTIC, which reports 29.9 BLEU-4 on MSCOCO and 19.4 BLEU-4 on MSCOCO→Flickr30k transfer SynTIC. We re-run it on the same caption corpus, encoder and decoder. SynTIC also uses feature projection and object tags from a DETR detector, so we either include both components in every arm or remove both from every arm and report both settings. Under equal tuning budgets, matching SynTIC within 0.5 BLEU-4 at no more than 10% of its generator calls counts as success.

Video setting. The reference is WSV at 33.9 BLEU-4 and 45.5 CIDEr on MSR-VTT, and 52.0 BLEU-4 and 95.7 CIDEr on MSVD WSV. Here the bar is retaining most of WSV's CIDEr while removing per-sample CogVideoX calls.

Key-ingredient ablations. The most important ablation trains the student directly on the same $K$ seed images, with no synthesizer. If amortizing gives nothing beyond the $K$ images themselves, the synthesizer is pointless. A second ablation keeps the generator architecture but drops the text condition, so it learns only the overall image-embedding distribution. This isolates the caption-dependent residual $r$.

Why not just…?

Why not just center the embeddings and add noise? This is the real threat. If the residual after centering is mostly unstructured, centering plus noise will match us at zero cost. I0T's gains are large: Flickr30k text-to-image retrieval rises from 67.10% to 76.30% with centering alone. But those are retrieval gains, and generation may depend on finer structure. Mind the Gap's finding that accuracy peaks at a non-zero gap suggests a constant shift leaves something on the table. The explained-variance diagnostic and the head-to-head comparison settle the question. If the synthesizer's margin over centering is under 1.5 BLEU-4, we report a negative result.

Why not just generate fewer Stable Diffusion images and train on those? That is exactly the seed-only ablation. Our bet is that one generated embedding per caption across the whole corpus beats real image features on only $K$ captions. SynTIC's report that it beats full-data baselines with 10% of the data shows that small synthetic budgets go far, which makes this ablation essential. We plot both arms against $K$. If the curves meet at small $K$, the synthesizer adds nothing.

Why not just tune LayerNorm or fine-tune the encoder? Tuning LayerNorm beats LoRA while training a tiny fraction of the parameters. It still needs paired multimodal training data and gradient access inside the model. Text-only training has neither pairs nor, for API-served encoders, gradients. We therefore treat it as out of scope rather than as a competitor. Where pairs do exist, adding it would be a different problem.

Why not just learn a WSV-style polisher? A polisher corrects embeddings that already exist, so it still requires per-sample generation. Training it on the $K$ seeds and applying it to shifted text embeddings is effectively a deterministic version of our synthesizer. We include it as a deterministic regressor arm. If it matches the diffusion synthesizer, the stochastic component is unnecessary.

Proposed Experiments

The main experiment trains on MSCOCO captions and evaluates on MSCOCO with BLEU-4, CIDEr and SPICE, and on MSCOCO→Flickr30k transfer. For every arm we report a quality-versus-generator-GPU-hours curve with $K$ ranging from 1k to 50k.

The first decisive check is whether, at equal generator cost, the synthesizer beats seed-only training. The second is whether, at near-zero cost, it beats centering plus noise. Both must hold.

A gap-distance sweep tests whether the best captioning gap is non-zero, as Mind the Gap's classification results would predict. A portability test reuses the synthesizer recipe unchanged with Long-CLIP and with a video-frame encoder on MSR-VTT and MSVD against WSV. The method claims to be encoder-agnostic, and this test checks that claim.

A final transfer probe asks whether the method's value is specific to captioning. On Amazon Baby and Clothing, we hide images for 30% of items and synthesize their image embeddings from item text. The resulting embeddings are fed to CLIPER-style FREEDOM and DualGNN backbones, and Recall@10 and Recall@50 are compared against centered text embeddings and against the real images.