FAME: Fairness-aware Attention-modulated Video Editing

Zhangkai Wu, Xuhui Fan, Zhongyuan Xie, Kaize Shi, Zhidong Li, Longbing CaoarXiv 2025 · arXiv · OpenReview

Abstract

Training-free video editing (VE) models tend to fall back on gender stereotypes when rendering profession-related prompts. We propose FAME for Fairness-aware Attention-modulated Video Editing that mitigates profession-related gender biases while preserving prompt alignment and temporal consistency for coherent VE. We derive fairness embeddings from existing minority representations by softly injecting debiasing tokens into the text encoder. Simultaneously, FAME integrates fairness modulation into both temporal self-attention and prompt-to-region cross-attention to mitigate the motion corruption and temporal inconsistency caused by directly introducing fairness cues. For temporal self-attention, FAME introduces a region-constrained attention mask combined with time decay weighting, which enhances intra-region coherence while suppressing irrelevant inter-region interactions. For cross-attention, it reweights tokens to region matching scores by incorporating fairness-sensitive similarity masks derived from debiasing prompt embeddings. Together, these modulations keep fairness-sensitive semantics tied to the right visual regions and prevent temporal drift across frames. Extensive experiments on a new VE fairness-oriented benchmark, FairVE, demonstrate that FAME achieves stronger fairness alignment and semantic fidelity, surpassing existing VE baselines.

Introduction: The Challenge of Fair Video Editing

Large-scale Diffusion Models, such as Stable Diffusion, frequently inherit social biases from their training data. In video editing, this manifests as gender stereotypes: a prompt for a "doctor" may default to a male representation, while a "nurse" defaults to female.

While debiasing techniques exist for static images, applying them to video is significantly more complex. Simply modifying the text prompt (e.g., from "doctor" to "female doctor") often triggers motion corruption. The resulting videos suffer from "ghosting," loss of subject tracking, or jittery frames because the model’s internal attention mechanisms struggle to align new fairness attributes with the original motion. FAME addresses this by balancing fairness with temporal consistency.

Comparison of FAME against baselines. FAME successfully corrects gender bias (e.g., changing a male teacher to a female teacher) without the motion corruption seen in naive debiasing approaches.

Methods: The FAME Framework

FAME introduces a three-pronged, training-free approach that operates entirely during the inference stage, ensuring efficiency and compatibility with existing pipelines.

  1. Soft Debiasing Prompt Encoding: Instead of a hard word swap, FAME blends fairness-aware embeddings and injects them into the End-of-Sequence (EOS) token. As the EOS token aggregates global semantics in diffusion text encoders, this allows the model to incorporate fairness intent without disrupting the local word-level alignments necessary for structural preservation.
  2. Temporal Self-Attention Modulation: To prevent motion corruption, FAME enforces "intra-region coherence." It utilizes a region-constrained attention mask and time-decay weighting to ensure that pixels belonging to the subject remain consistent across frames while suppressing irrelevant background interactions.
  3. Cross-Attention Reweighting: This module ensures fairness-related tokens (e.g., "female") map to correct spatial regions. By using cosine similarity to reweight attention scores, FAME prevents "semantic drift," where fairness attributes might erroneously apply to the background.
The FAME architecture, illustrating the Soft Debiasing Prompt Encoding, Self-Attention Manipulation, and Cross-Attention Reweighting modules.

Datasets: The FairVE Benchmark

The authors introduced FairVE, the first dataset specifically designed to evaluate fairness in video editing. It consists of 640 video-text pairs covering 8 representative professions (CEO, doctor, technician, pilot, teacher, nurse, fashion designer, and librarian) across two genders. Videos were curated from the DAVIS benchmark and high-quality web clips, filtered to ensure a single human subject for clear evaluation.

Key Results

FAME was rigorously tested against state-of-the-art baselines like TokenFlow and VideoGrain, yielding the following insights:

  • Fairness Alignment: FAME achieved significantly higher "Correction Ratios." In human evaluations, its Bias Correction scores frequently exceeded 80/100.
  • Temporal Consistency: FAME achieved the lowest Warp-Err (1.66), a metric for video jitter. Baselines often exhibited errors nearly double this value (up to 3.08).
  • Semantic Fidelity: FAME outperformed all baselines in CLIP-F (frame consistency) and TIFA (text-to-image alignment) scores, proving that fairness does not require sacrificing video quality.

Limitations and Future Work

While FAME represents a significant step forward, it is currently optimized for videos with a single human subject. Handling complex scenes with multiple actors or highly dynamic backgrounds remains an open challenge. Furthermore, while this study focuses on gender-profession stereotypes, the authors acknowledge the need to expand this framework to address racial, age, and intersectional biases in future research.

Summary updated

Abstract

Training-free video editing (VE) models tend to fall back on gender stereotypes when rendering profession-related prompts. We propose \textbf{FAME} for \textit{Fairness-aware Attention-modulated Video Editing} that mitigates profession-related gender biases while preserving prompt alignment and temporal consistency for coherent VE. We derive fairness embeddings from existing minority representations by softly injecting debiasing tokens into the text encoder. Simultaneously, FAME integrates fairness modulation into both temporal self attention and prompt-to-region cross attention to mitigate the motion corruption and temporal inconsistency caused by directly introducing fairness cues. For temporal self attention, FAME introduces a region constrained attention mask combined with time decay weighting, which enhances intra-region coherence while suppressing irrelevant inter-region interactions. For cross attention, it reweights tokens to region matching scores by incorporating fairness sensitive similarity masks derived from debiasing prompt embeddings. Together, these modulations keep fairness-sensitive semantics tied to the right visual regions and prevent temporal drift across frames. Extensive experiments on new VE fairness-oriented benchmark \textit{FairVE} demonstrate that FAME achieves stronger fairness alignment and semantic fidelity, surpassing existing VE baselines.
BibTeX
@misc{wu2025fame,
  author = {Zhangkai Wu and Xuhui Fan and Zhongyuan Xie and Kaize Shi and Zhidong Li and Longbing Cao},
  title = {{FAME: Fairness-aware Attention-modulated Video Editing}},
  year = {2025},
  eprint = {2510.22960},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.22960},
}
Figure