Does a Personalized Subject Survive a Change of Language?

Subject-driven image generation lets a user show a text-to-image model three to five photos of their dog, child, or product and then ask for "my dog surfing" or "my dog in the snow." Every widely used method does this by teaching the model a new word, a rare token such as [V], paired with an English class noun. Most of the world does not prompt in English, and multilingual image generators now exist. So a practical question has gone untested: if a subject is learned from English captions, does the model still render that subject when the user writes the prompt in Chinese, Hindi, or Arabic? Or does the new word stop working once its English neighbors are gone?

This note proposes to measure that failure and to fix it cheaply. We suspect the personalized token does not bind to the subject by itself. It binds to the subject in English context, because a multilingual text encoder processes each token in light of its neighbors. Our fix adds one inexpensive term during personalization. It makes the encoder's output at the subject token stay the same across short template sentences in a few source languages. We then test whether this invariance carries over, zero-shot, to languages never seen during personalization. That is the same property the cross-lingual transfer literature relies on for text tasks.

Core Claim

The claim has three parts, and each can fail on its own terms. The first is that English-trained personalization on a multilingual diffusion backbone loses subject fidelity when prompted in other languages, and loses more in non-Latin scripts. A diffusion model is a generator that learns to turn random noise into images step by step. The second is that the loss comes mainly from the text side: the identifier's contextual encoding drifts with the surrounding language. It does not come from the image generator failing to understand the target-language scene. The third is that a template-level invariance loss, applied in a handful of source languages, closes most of that gap for held-out languages. It should cost far less than translating the training captions, and it should beat translating the prompt at inference time on prompts whose meaning is culturally specific.

The file contains no prior work that personalizes a subject and then prompts it across languages. The closest multilingual generation work stays at the class level. AltDiffusion replaces Stable Diffusion's English text encoder with a distilled XLM-R encoder and retrains the cross-attention key and value projections, which are the layers that connect words to image regions. It never binds an individual subject. Translation-Enhanced Multilingual Text-to-Image Generation fuses a native-language prompt with several machine translations through a small adapter, but only for generic prompts. The benchmarks CoCo-CroLa and LingT2I motivate this proposal. They document that the same prompt produces different concepts, cultural stereotypes, and collapse under token fragmentation across languages. For example, LingT2I reports SD3.5 content-generation scores of 0.79 in English against 0.24 in Hindi. Both benchmarks test only dictionary nouns and generic scenes, not a subject the user owns.

Research Program

The first stage is diagnostic, and the rest of the program depends on it. We build a Multilingual DreamBench from the 30 subjects and 25 prompts per subject in the original DreamBooth benchmark. We add the human subjects from DreamBench++, because that paper shows identity drift is worst on faces and poorly captured by DINO and CLIP. Prompts are professionally translated into languages the backbone supports, chosen to span Latin scripts (Spanish, French) and non-Latin scripts (Chinese, Japanese, and Arabic, Hindi, or Korean where the backbone covers them). We add a second, culture-bound prompt set whose scenes come from the six categories of AltDiffusion's MC-18 (festivals, clothing, food, landmarks, painting, literature). Written natively in each language, it asks, for instance, for the subject at a named festival or in a named painting style.

Because the prompts are parallel across languages, evaluation stays clean. Concept preservation compares generated images to the reference photos and does not depend on language. We score it with DINO, with a face-recognition cosine similarity for human subjects (the identity metric used by MyStyle), and with the GPT-4o judge protocol of DreamBench++. Prompt following is judged against the English source of each prompt. This avoids the English bias of CLIP-based text scores on foreign-language text. On the culture-bound set, it is complemented by native-speaker ratings. We also report a cross-lingual consistency score: image similarity between generations of the same prompt, with the same seed, across languages.

On this benchmark we personalize AltDiffusion with DreamBooth and with a LoRA variant. LoRA inserts small trainable low-rank matrices into frozen layers. We measure the per-language gap and test which side causes it. One probe swaps the target-language subject phrase for the English one while keeping the target-language scene words. If fidelity recovers, the text-side hypothesis holds. We also correlate the gap with token-per-character ratios, following LingT2I's finding on fragmentation. A separate check on human subjects asks whether non-English prompts pull identity toward culturally stereotyped faces. This is a personalized version of the "generic collisions" that CoCo-CroLa reports.

The second stage is the method. Let $E$ be the frozen multilingual text encoder, $v$ the learnable input embedding of the identifier, $t_\ell$ a short template such as "a photo of [V] dog" written in language $\ell$ from a small source set $S$, and $h_\ell(v)$ the encoder output at the identifier and class-noun positions. We add

$$ \mathcal{L}_{\text{inv}}(v) = \sum_{\ell \in S} \big\| h_\ell(v) - h_{\text{en}}(v) \big\|^2 $$

to the usual DreamBooth denoising and class-prior preservation losses, with weight $\lambda$. Only the class-noun template needs translating, which is a dictionary-level cost. The loss requires encoder forward passes but no extra denoising passes through the image generator. This is why it should be much cheaper than training on translated captions.

The method reuses two ideas from the transfer literature. Crosslingual Generalization through Multitask Finetuning shows that task behavior learned in a few languages carries over to many. The reading-comprehension transfer work Zero-shot Reading Comprehension by Cross-lingual Transfer Learning shows that preserving the pretrained cross-lingual geometry matters more than surface translation. The strict test is on languages outside $S$, which receive no personalization signal at all. We also try a cross-lingual variant of prior preservation, with class samples generated from the class noun in each source language. This probes whether an English-only prior anchor lets the class concept drift in other languages, as CoCo-CroLa's collisions suggest it might.

Baselines

The naive reference point is English DreamBooth on AltDiffusion, prompted directly in the target language. It tells us whether a gap exists at all. The same model prompted in English is the upper reference for every language. For orientation only, DreamBooth reports DINO 0.668 on DreamBench with English Stable Diffusion, and DisenBooth reports DINO 0.685 for DreamBooth and 0.675 for itself, with CLIP-T 0.330 for DisenBooth. These are English, non-multilingual numbers. No reported result was found on Lacuna for any personalization method under non-English prompts, so our own English-prompt run on the same backbone is the reference.

The strongest competitor is translate-at-inference. The target-language prompt is machine-translated to English, keeping [V] intact, and passed to the same English-personalized model. We also run a stronger multi-translation version in the spirit of ENSAD, averaging the embeddings of several sampled translations. ENSAD's paper reports FID 14.52 on COCO-CN against 15.57 for plain zero-shot prompting, though for generic generation with a GAN, not personalization. Translate-at-inference is the strongest competitor because DreamBench prompts are short and literal, so translation should rarely fail on them. Beating it means matching it within noise on the literal set and exceeding it on prompt following and native-speaker ratings for the culture-bound set. It also means doing this at a fraction of the inference latency; mCLIP measured 438.7 ms for translate-then-retrieve against 17.2 ms for native encoding.

Translate-train fine-tunes on reference images paired with captions translated into every source language. We run it under a matched budget of wall-clock time and denoising steps, and separately with unmatched full compute as a ceiling. Beating it means comparable held-out-language fidelity at matched compute. DisenBooth on the same backbone tests whether the problem is really background entanglement rather than language. The key ablation sets $\lambda = 0$ while keeping the multilingual templates in the input. This isolates the invariance term from mere exposure to foreign-language templates. All methods share subjects, seeds, number of generated images, backbone, and a fixed hyperparameter search budget tuned only on English and source-language validation prompts.

Why not just…?

Why not just translate the prompt to English at inference? This is a real threat, and on literal prompts it may simply win. The case against it rests on meaning that translation discards. AltDiffusion shows the painter "Baishi Qi" translating to "white stone" and reports that native multilingual generation beat translation-based Stable Diffusion in human culture-consistency ratings across all 18 of its languages. There is also a latency argument. The culture-bound prompt set exists to test exactly this. If translation matches us there too, we will report that cross-lingual personalization reduces to a translation problem, which is itself a useful finding.

Why not just fine-tune on translated captions? It multiplies the expensive denoising passes by the number of languages. The ENSAD paper found translate-train beaten by a lightweight adapter even for generic generation. It also bakes translation errors into the subject's weights. Our matched-compute comparison settles whether the invariance loss buys the same held-out-language fidelity for less. If translate-train wins even at matched compute, the invariance idea is not worth its complexity.

Why not just personalize a native multilingual model and trust it? That is our naive baseline, and it may already work for Latin-script languages. LingT2I's Hindi collapse and its fragmentation analysis suggest it will not hold uniformly, but that is a prediction, not evidence about personalized subjects. The diagnostic stage measures it directly. If the gap is small everywhere, the method stage is dropped and the benchmark and the identity-drift analysis become the contribution.

Decisive Checks

Three results decide the project. The first is whether English-personalized subjects lose concept preservation in non-English prompts, and whether the swap probe shows the loss comes from the subject phrase rather than the scene words. If neither holds, there is no text-side problem to fix. The second is whether the invariance loss, trained with a source set of perhaps three languages, recovers fidelity on held-out languages, especially non-Latin scripts, beyond the $\lambda = 0$ ablation. Without that, the "zero-shot" claim fails. The third is whether it beats translate-at-inference on the culture-bound set by native-speaker judgment while staying level on literal prompts. Without that, the practical case collapses to "use a translator." We will also report face-identity similarity per language for human subjects, because a subject that remains recognizable in English but acquires stereotyped features in Hindi would be the most consequential failure this study could expose.