AlignVid: Training-Free Attention Scaling for Semantic Fidelity in Text-Guided Image-to-Video Generation

Yexin Liu, Wen-Jie Shu, Zile Huang, Haoze Zheng, Yueze Wang, Jingjin Zhu, Manyuan Zhang, Ser-Nam Lim, Harry Yang

TL;DR

Text-guided image-to-video (TI2V) models often ignore text prompts that require major structural changes to a reference image due to a phenomenon called "visual dominance." To fix this, the authors introduce AlignVid, a training-free method that scales internal attention representations to reduce attention entropy and amplify text signals. Paired with a new evaluation benchmark (OmitI2V), AlignVid demonstrates significant improvements in semantic fidelity across multiple video generation architectures with near-zero computational overhead.

What Problem Does This Paper Address?

When generating a video from a reference image and a text prompt, current state-of-the-art models excel at animating the scene but frequently fail when the prompt demands structural edits—like adding, deleting, or modifying objects.

The authors identify the root cause as visual dominance: the high-fidelity reference image overwhelms the model's attention mechanism. This causes severe attention dispersion (high entropy) across the video frames, effectively drowning out the text prompt's instructions and causing the model to simply replicate the static reference image.

Baseline models often fail to add or modify objects as requested by the text prompt, while AlignVid successfully enforces semantic adherence.

Methods

The authors propose AlignVid, a plug-and-play, training-free intervention consisting of two main components:

  1. Attention Scaling Modulation (ASM): Instead of modifying the input image, ASM mathematically adjusts the temperature of the softmax function within the attention layers by scaling the Query ($Q$) or Key ($K$) representations. This sharpens the attention distribution, forcing the model to focus on semantically dense text tokens rather than redundant visual features.
  2. Guidance Scheduling (GS): Applying ASM indiscriminately can degrade visual quality. GS solves this via two-level gating:
  3. Block-level Guidance Scheduling (BGS): Restricts modulation only to "foreground-sensitive" transformer blocks, identified via a lightweight calibration process using PCA and an off-the-shelf grounding model.
  4. Step-level Guidance Scheduling (SGS): Restricts modulation to early denoising steps where global semantic layout is established, leaving later steps free to refine fine-grained aesthetic details.

Datasets

  • OmitI2V: A newly introduced benchmark specifically designed to evaluate fine-grained semantic fidelity in TI2V models. It contains 367 human-annotated image-text pairs across three edit types: modification (34.19%), addition (33.16%), and deletion (32.65%). It spans diverse domains like living beings, structures, and arts.
  • VBench, GenEval, and ImgEdit: Used to evaluate the generalization of AlignVid to standard text-to-video, text-to-image, and image editing tasks.

Key Insights and Ideas

  • The Blur Paradox: In a pilot study, the authors found that applying a slight Gaussian blur to the input image counter-intuitively improved the model's adherence to the text prompt and increased motion. Blurring removes low-level visual noise, which lowers attention entropy and "liberates" the text prompt from visual dominance.
  • Attention as an Energy Landscape: By viewing attention through an energy-based lens, the authors realized that scaling $Q$ or $K$ acts as an inverse-temperature control knob. This allows them to replicate the entropy-reducing benefits of blurring entirely inside the model, avoiding any input-level image degradation.
Figure 2. Pilot example. (a) Videos and attention maps generated from the original input image (top) and from the same image after applying a Gaussian blur (bottom). (b) Attention Visualization: In the original setting, the model exhibits visual dominance, where excessive focus on the reference image suppresses textual constraints and temporal dynamics. Applying blur weakens this dominance, shifting attention toward text tokens and temporal neighbors. (c) Quantitative Analysis: Statistics across 30 samples reveal that applying blur leads to a universal increase in attention scores and a universal reduction in entropy across image, video, and text modalities.

Key Results

  • Substantial Semantic Gains: Integrating AlignVid into state-of-the-art models like FramePack and Wan2.1 yielded consistent improvements in semantic fidelity. For example, on Wan2.1, it boosted modification accuracy by 4.85% and addition accuracy by 7.79% on the OmitI2V benchmark.
  • Negligible Overhead: Because it only scales existing tensors during specific steps and blocks, AlignVid introduces virtually zero computational overhead, adding only +0.01% to +0.09% to the total runtime.
  • Cross-Domain Generalization: AlignVid also proved effective outside of TI2V, showing performance gains when applied to text-to-image (GenEval) and image editing (ImgEdit) tasks.

Limitations and Future Work

  • Semantics-Aesthetic Trade-off: Strongly enforcing prompt adherence can slightly reduce aesthetic quality scores (e.g., a minor drop in Wan2.1's aesthetic score from 63.12 to 61.63).
  • Motion Blur: When AlignVid successfully triggers large temporal changes and stronger motion, the resulting videos can occasionally exhibit mild motion blur, which negatively impacts perceived sharpness metrics.
  • Future Directions: The authors suggest exploring more sophisticated, automated ways to dynamically balance this trade-off per token or step without manual tuning of the scaling factor.

Summary updated

Abstract

Text-guided image-to-video generation has made substantial progress, yet it still struggles to execute text-specified edits that require substantial changes to a reference image (\textit{e.g., object addition, removal, or modification}). Empirically, our analysis reveals that this stems from \textbf{visual dominance}, where the reference image causes severe attention dispersion, inhibiting the model's ability to incorporate new semantic information. To address this, we propose \textbf{AlignVid}, a training-free intervention that re-calibrates the model's internal attention distribution. Drawing on an energy-based perspective of attention, AlignVid employs Attention Scaling Modulation (\textbf{ASM}) to reduce attention entropy and concentrate focus on semantic tokens, alongside Guidance Scheduling (\textbf{GS}) to maintain generation stability. To rigorously assess this capability, we present \textbf{OmitI2V}, a comprehensive benchmark for evaluating prompt adherence across object modification, addition, and deletion. Extensive experiments demonstrate that AlignVid effectively enhances semantic fidelity with negligible computational overhead. Code and the OmitI2V benchmark are available at https://github.com/LAW1223/AlignVid.
BibTeX
@inproceedings{liu2026alignvid,
  author = {Yexin Liu and Wen-Jie Shu and Zile Huang and Haoze Zheng and Yueze Wang and Jingjin Zhu and Manyuan Zhang and Ser-Nam Lim and Harry Yang},
  title = {{AlignVid: Training-Free Attention Scaling for Semantic Fidelity in Text-Guided Image-to-Video Generation}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=bgtUrDtKjF},
}
Figure