Text-to-image diffusion models generate a picture by starting from random noise and removing it a little at a time, over dozens of passes through a large neural network. The main way to make them cheap enough for interactive use is step distillation: a slow "teacher" sampler is compressed into a "student" that produces an image in one to four passes. Separately, a cheap route to multilingual image generation now exists. MuLan trains a small adapter on English image-caption pairs only. The adapter sits between a frozen multilingual text encoder and a frozen image generator, and prompts in 110+ languages then work "for free", because the encoder already places "dog" and "perro" near each other. This is zero-shot cross-lingual transfer: the model never sees a target-language training image. The two cost reductions have never been studied together. The question is whether a few-step student, distilled on English prompts, keeps the teacher's multilingual ability. Or does it quietly become an English-only fast model with a slow multilingual fallback?
We expect the latter, and we expect the loss to be uneven. The student is trained to reproduce the teacher's trajectories for the narrow distribution of English prompt embeddings. Non-English embeddings are close to that distribution but not on it, and the gap is largest for languages the encoder splits into many subword pieces. LingT2I already ties this fragmentation to worse image quality. A many-step sampler reads the conditioning dozens of times and can partly correct a slightly-off embedding; a one-step student gets one chance. We propose to measure this effect, locate where along the denoising trajectory it arises, and remove it. The fix is to distill the student on target-language prompts against the teacher's own trajectories. Trajectory distillation needs only prompts and noise, not images, so the fix keeps the "no target-language images" property that makes zero-shot transfer cheap.
Core Claim
Our claim is that step distillation does not keep cross-lingual parity. With English-only distillation, the gap between English and target-language prompt fidelity should grow as the number of sampling steps shrinks. The growth should be steepest for non-Latin, highly fragmented languages such as Hindi and Arabic. We further claim the damage can be reversed at no extra distillation compute by changing only which prompts the distillation loop sees. Two variants make this concrete. In self-teacher distillation, the teacher is conditioned on the target-language prompt itself; this needs only unlabeled target-language text and is fully zero-shot. In pivot-teacher distillation, the teacher is conditioned on an English caption while the student is conditioned on a parallel target-language version of it. The parallel version is produced by machine translation at training time only, so the student can in principle beat its own teacher on target languages.
The second variant borrows the central insight of Images as Pivots. That work freezes Stable Diffusion and trains a Chinese text encoder so that cross-attention maps inside the image network match those produced by English captions. Its lesson is that alignment must happen at the level of what the image network actually consumes, not at the level of pooled sentence vectors. We move the matching target from cross-attention maps to the full denoising output of a few-step student. We need no images: the frozen teacher supplies the target trajectory. We also train neither the text encoder nor the backbone, only the student's distillation parameters.
Where This Sits
Prior work on multilingual diffusion falls into three groups, and all three leave sampling cost untouched. AltDiffusion distills OpenCLIP into XLM-R and then fine-tunes the whole image network on multilingual data. Images as Pivots handles Chinese with 3M translated pairs. MuLan achieves English-only zero-shot transfer in about 96 A100-hours. MuLan states that its adapter is compatible with fast sampling modules, but the file gives no measurement of multilingual fidelity under few-step sampling. That missing measurement is the gap this proposal targets.
On the efficiency side, step reduction through conditional flow matching and consistency models reaches one to four steps, but these methods are developed and judged in English. Early-Bird Diffusion supplies an analysis lens rather than a method. It shows that denoising steps fall into regimes of different importance: early, high-noise steps fix global structure, and late steps refine texture. It exploits this for timestep-aware pruning in unconditional and class-conditional models. We use the same regime split to ask in which phase the language gap opens.
Two papers constrain the design. Absorbing Quantization Error shows that per-step errors build up across the trajectory and distort intermediate states. This is the same compounding we suspect off-manifold embeddings suffer. pplx-embed shows that aggressive compression holds up at large capacity but collapses at small capacity. This warns that a low-rank distillation adapter may be too small to absorb many languages, so we vary its rank.
The motivating evidence comes from the cross-lingual transfer literature. MLQA and Hsu et al. show that shared multilingual representations transfer well but unevenly. LingT2I shows the same unevenness in image generation, finding that adapter models like MuLan buy lower cross-lingual variance at some cost in English peak quality. Our hypothesis is that distillation is a new place where this unevenness can grow.
Research Program
The program has three stages.
The first stage is a measurement study. We take MuLan-style systems on SD 1.5 and SDXL backbones and pair each with standard English-only latent consistency distillation. We then sweep the number of sampling steps from the full teacher schedule down to one. At each step count we score all ten LingT2I languages and a subset of Crossmodal-3600 languages. The quantity of interest is the ratio of target-language fidelity to English fidelity. We regress this ratio, at each step count, on each language's token-to-word ratio. If the slope does not steepen as steps fall, the core claim is wrong. In that case the paper becomes a useful negative result: zero-shot transfer survives distillation.
The second stage is a diagnostic that borrows the regime split from Early-Bird Diffusion. In the many-step teacher, we condition on the English prompt for some regimes and on the parallel target prompt for the others, and measure which regime accounts for most of the fidelity loss. If the high-noise regime dominates, as the structure-versus-texture account predicts, we weight the distillation loss toward those timesteps.
The third stage is the method itself. Self-teacher and pivot-teacher distillation are each run with the same optimizer, step budget and adapter rank as the English-only distillation. Only the prompt mixture changes.
Baselines
The naive reference point is the many-step teacher itself, MuLan with its full sampler. It sets the multilingual fidelity we want to keep, and its latency sets the cost we want to cut. On translated COCO2014, MuLan reports an average non-English CLIP similarity of 39.61 against 39.57 for English. We will rerun it under our own prompts rather than rely on that figure.
The strongest efficient competitor is MuLan combined with off-the-shelf English latent consistency distillation, run at one, two and four steps. This is what a practitioner would deploy today, and MuLan advertises the compatibility. We found no reported result on Lacuna for multilingual fidelity of MuLan under few-step consistency sampling, so we will produce the number ourselves.
The second strong competitor is translate-at-inference: a machine-translation model feeds a few-step English student. We will charge it the translation latency.
For Chinese specifically, we will compare against Images as Pivots. It reports FID 13.43 on MS-COCO, against 18.14 for AltDiffusion and 17.09 for Taiyi, so we report Chinese FID on the same protocol.
All distilled variants share the teacher, the English caption pool (a LAION-EN aesthetic subset like MuLan's), GPU-hours, adapter rank and hyperparameter search budget. Our method beats the strongest competitor if it does two things. At four steps or fewer, it must recover at least half of the target-to-English fidelity gap that English-only distillation opens on Hindi and Arabic. And it must lose no more than one point of English CLIP similarity.
Two ablations isolate the key ingredient. The first matches the number of distillation prompts but draws them all from English. This rules out "more prompts" as the explanation. The second is pivot-teacher versus self-teacher. It separates the gain from merely seeing target-language embeddings from the gain of an English teacher signal.
Why not just…?
Why not just translate the prompt to English and use a fast English model? This is the most serious threat. If translation is good, it may match our method on everyday prompts. The evidence against it is partial. AltDiffusion shows literal translation erasing cultural referents, for example a painter's name rendered as "white stone", and native multilingual generation beats translation-based Stable Diffusion on the MC-18 cultural benchmark. Hsu et al. found that machine-translated data underperformed direct zero-shot transfer in reading comprehension. Translation also adds a sequential model call to a pipeline whose whole point is low latency. We will settle the question empirically on two prompt sets. On the general DOCCI-derived LingT2I prompts we expect translation to be competitive. On MC-18 cultural prompts we expect it to lose. We will report end-to-end latency for both pipelines. If translation wins on both prompt sets at comparable latency, we will say so.
Why not just distill on English and accept the small loss? Whether the loss is small is exactly what stage one measures. The answer is unknown, and MuLan's compatibility claim is untested on this point. If the English-only student's target-to-English ratio stays within a point of the teacher's at one step, this alternative wins and the method is unnecessary.
Why not just run more steps for non-English prompts? Adaptive step counts keep quality but spend the latency saving precisely on the languages that are already underserved. This is the inequity we want to avoid. We include it as a cost-quality curve: if our four-step student matches an eight-step English-only student on Hindi, the method buys back half the latency.
Why not fine-tune the text encoder or adapter on target-language images, as AltDiffusion does? For most languages those images do not exist. That is the premise of zero-shot transfer, and IAP and MuLan both stress this scarcity. Our method touches no target-language images at all.
Proposed Experiments
The decisive check is the stage-one sweep on LingT2I content prompts, scored with its MetaCLIP2-based content metric. The paper's own reference points are SD3.5 at 0.79 for English and 0.24 for Hindi, which shows how wide gaps can be. We plot the target-to-English ratio against step count for every language and test whether fragmentation predicts the slope.
Beyond that check, we run four more evaluations. On Crossmodal-3600, whose automatic metrics correlate above 0.88 with human judgment across its languages, we report CLIP similarity per language for all distilled variants. On translated COCO2014 we report CLIP similarity, for direct comparison with MuLan's 39.61. On MC-18 we run human pairwise preference against translate-at-inference. For Chinese we report MS-COCO FID against IAP's 13.43.
Cross-lingual variance, the summary statistic LingT2I uses, is reported at each step count. This shows whether distillation pushes MuLan's fairness advantage toward the high-variance end where LingT2I places native English-centric models.
The diagnostic swaps conditioning between English and target languages within one regime at a time and reports per-regime fidelity loss. Finally, adapter rank is swept at three sizes. Following the pplx-embed warning about small-capacity collapse, this tests whether multilingual distillation needs more student capacity than English-only distillation does.
References
- MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
- On the Limitations of Cross-Lingual Consistency in Multilingual Text-to-image Generation
- Efficient Cross-Lingual Transfer for Chinese Stable Diffusion with Images as Pivots
- AltDiffusion: A Multilingual Text-to-Image Diffusion Model
- Zero-Shot Voice Conversion via Content-Aware Timbre Ensemble and Conditional Flow Matching
- Multimodal Image Synthesis and Editing: The Generative AI Era
- Early-Bird Diffusion: Investigating and Leveraging Timestep-Aware Early-Bird Tickets in Diffusion Models for Efficient Training
- Absorbing Quantization Error by Deformable Noise Scheduler for Diffusion Models
- Memory-Efficient Multilingual Embeddings with a Diffusion-LM Backbone
- MLQA: Evaluating Cross-lingual Extractive Question Answering
- Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model
- Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset