TokenStream: Multi-Subject Personalized Video Generation with Token-wise LoRA and Guided Attention

Hung Nguyen, Quang Qui-Vinh Nguyen, Khoi NguyenarXiv 2026 · arXiv

TL;DR

TokenStream is a new framework for generating videos featuring multiple personalized subjects (like your specific dog and a favorite teapot) using only a single reference image for each. By isolating subject updates to specific text tokens and using a "guided stream" to borrow motion priors from a base model, it avoids the common pitfalls of identity mixing and "static-image bias" where personalized subjects forget how to move.

What Problem Does This Paper Address?

Personalizing text-to-video (T2V) models—making them generate a specific person or object—is notoriously difficult, especially when multiple subjects are involved. Existing methods like DreamBooth or Textual Inversion (TI) often struggle in two ways:

  1. Identity Mixing: When you try to combine independently trained adapters for a "dog" and a "teapot," the features often bleed into each other, resulting in a dog-shaped teapot or a teapot-colored dog.
  2. Static-Image Bias: Because these models are often fine-tuned on a single static reference image, they tend to "overfit" to that stillness. The resulting videos often feature rigid, stationary subjects that fail to follow the motion described in the text prompt.

The authors aim to create a system where you can train an adapter for a subject once and then "plug and play" it into any multi-subject scene without further optimization or losing the base model's natural motion.

Figure 1. Multi-subject video personalization with TokenStream. TokenStream uses a single reference image per subject and composes independently trained subject adapters at inference time. Compared with Textual Inversion and DreamBooth-style adaptation, it better preserves subject appearance while retaining the motion and prompt-following behavior of the pretrained T2V model.

Methods

TokenStream introduces two primary technical innovations built on top of the Wan2.1 diffusion transformer (DiT) architecture:

1. Token-wise LoRA (ToRA)

Standard LoRA (Low-Rank Adaptation) updates the weights of the entire model. ToRA, however, is surgical. It applies low-rank updates only to the cross-attention "Value" (V) projections associated with specific text tokens.

  • How it works: If the prompt is "A dog plays with a teapot," the "dog" LoRA is only active when the model processes the "dog" token, and the "teapot" LoRA is only active for the "teapot" token.
  • Why it matters: This creates a separate adaptation path for every subject, allowing them to be composed naturally at inference time without interfering with one another.

2. T2V-Guided Stream

To fight the "static-image bias," the authors use a dual-stream approach during the early stages of video generation (denoising).

  • The Mechanism: They run an auxiliary "guided stream" using the frozen, un-personalized base model alongside the personalized "main stream."
  • The Trick: They copy the Query (Q) and Key (K) matrices from the base model to the personalized model. In attention mechanisms, Q and K define the structure and layout (the "where" and "how" of motion), while V defines the content (the "what" or appearance).
  • Result: The video inherits the sophisticated motion and layout priors of the base T2V model while injecting the personalized appearance learned by ToRA.
The overall TokenStream framework showing the dual-stream guidance and token-wise LoRA routing.

Datasets

The researchers evaluated their method on two benchmarks:

  • OpenS2V-Eval: A standard benchmark for subject-driven video.
  • Multi-DB (New): Introduced in this paper, Multi-DB contains 30 subjects and 122 prompts. Unlike existing benchmarks, it uses "appearance-free" prompts (e.g., "A dog jumps" instead of "A brown fluffy dog jumps"). This forces the model to rely entirely on the reference image for identity, providing a much stricter test of personalization.

Key Insights and Ideas

  • Decoupling Layout from Identity: The most significant "aha moment" is that you don't need to transfer the entire attention state to get good motion. By only transferring Q and K, you preserve the spatial relationships and temporal flow of the base model without overwriting the personalized visual features in V.
  • The Power of Token-Level Routing: Moving personalization from the "model level" to the "token level" solves the compositionality problem. It turns subject adapters into modular components that don't require joint training to work together.
  • Prompt Leakage is Real: The authors highlight that many existing benchmarks "cheat" by describing the subject's appearance in the prompt. Their Multi-DB benchmark reveals that many models rely on these text cues rather than actually learning the visual identity from the reference image.

Key Results

TokenStream demonstrates a clear quantitative edge over optimization-based baselines:

  • Identity Preservation (IP): On the Multi-DB benchmark (1.3B model), TokenStream achieved an IP score of 0.467, significantly higher than DreamBooth (0.422) and Textual Inversion (0.347).
  • Textual Alignment (TA): It also followed prompts better, with a TA score of 0.688 compared to DreamBooth's 0.652.
  • Efficiency: Training a ToRA adapter takes only about 4 minutes on an H100 GPU and results in a tiny 5.6 MB file, making it highly practical for real-world use.
Qualitative results showing successful multi-subject composition and interaction.

Limitations and Future Work

Despite its strengths, TokenStream has notable limitations:

  • Human Identities: The authors candidly admit the method does not yet reliably preserve human facial identities, likely due to the high sensitivity of the human eye to facial deviations and the limited viewpoint supervision from a single image.
  • Same-Class Subjects: If you try to generate two different dogs, the model tends to fuse them into a single identity because they share the same base "dog" token.
  • Shape Mismatch: If a reference subject's geometry differs wildly from the base model's prior (e.g., a very strangely shaped robot), the guided stream can sometimes bias the output back toward a more "generic" shape.

Future work will likely focus on incorporating multi-view reference images and developing better strategies for separating subjects of the same class.

Summary updated

Abstract

We introduce TokenStream, an optimization-based framework that composes independently learned subject identities in text-to-video generation without a reference-video training dataset or joint optimization of each new subject combination. Learning from a single reference image makes subject appearance difficult to separate from static-image bias, while merging independently trained adapters can mix identities. Token-wise LoRA (ToRA) restricts low-rank updates to the cross-attention value projections of subject-associated text tokens, providing a separate adaptation path for each subject. A T2V-guided stream copies queries and keys from the frozen base model during early denoising to reuse its layout and motion priors while retaining personalized values. We also introduce Multi-DB, a 30-subject, 122-prompt benchmark with appearance-free prompts and varied interactions. On Multi-DB at 1.3B, TokenStream improves identity preservation from 0.422 for DreamBooth to 0.467 and textual alignment from 0.652 to 0.688.
BibTeX
@misc{nguyen2026tokenstream,
  author = {Hung Nguyen and Quang Qui-Vinh Nguyen and Khoi Nguyen},
  title = {{TokenStream: Multi-Subject Personalized Video Generation with Token-wise LoRA and Guided Attention}},
  year = {2026},
  eprint = {2611.16897},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16897},
}
Figure