Abstract
Training-free video editing (VE) models tend to fall back on gender stereotypes when rendering profession-related prompts. We propose FAME for Fairness-aware Attention-modulated Video Editing that mitigates profession-related gender biases while preserving prompt alignment and temporal consistency for coherent VE. We derive fairness embeddings from existing minority representations by softly injecting debiasing tokens into the text encoder. Simultaneously, FAME integrates fairness modulation into both temporal self-attention and prompt-to-region cross-attention to mitigate the motion corruption and temporal inconsistency caused by directly introducing fairness cues. For temporal self-attention, FAME introduces a region-constrained attention mask combined with time decay weighting, which enhances intra-region coherence while suppressing irrelevant inter-region interactions. For cross-attention, it reweights tokens to region matching scores by incorporating fairness-sensitive similarity masks derived from debiasing prompt embeddings. Together, these modulations keep fairness-sensitive semantics tied to the right visual regions and prevent temporal drift across frames. Extensive experiments on a new VE fairness-oriented benchmark, FairVE, demonstrate that FAME achieves stronger fairness alignment and semantic fidelity, surpassing existing VE baselines.
Introduction: The Challenge of Fair Video Editing
Large-scale Diffusion Models, such as Stable Diffusion, frequently inherit social biases from their training data. In video editing, this manifests as gender stereotypes: a prompt for a "doctor" may default to a male representation, while a "nurse" defaults to female.
While debiasing techniques exist for static images, applying them to video is significantly more complex. Simply modifying the text prompt (e.g., from "doctor" to "female doctor") often triggers motion corruption. The resulting videos suffer from "ghosting," loss of subject tracking, or jittery frames because the model’s internal attention mechanisms struggle to align new fairness attributes with the original motion. FAME addresses this by balancing fairness with temporal consistency.
Methods: The FAME Framework
FAME introduces a three-pronged, training-free approach that operates entirely during the inference stage, ensuring efficiency and compatibility with existing pipelines.
- Soft Debiasing Prompt Encoding: Instead of a hard word swap, FAME blends fairness-aware embeddings and injects them into the End-of-Sequence (EOS) token. As the EOS token aggregates global semantics in diffusion text encoders, this allows the model to incorporate fairness intent without disrupting the local word-level alignments necessary for structural preservation.
- Temporal Self-Attention Modulation: To prevent motion corruption, FAME enforces "intra-region coherence." It utilizes a region-constrained attention mask and time-decay weighting to ensure that pixels belonging to the subject remain consistent across frames while suppressing irrelevant background interactions.
- Cross-Attention Reweighting: This module ensures fairness-related tokens (e.g., "female") map to correct spatial regions. By using cosine similarity to reweight attention scores, FAME prevents "semantic drift," where fairness attributes might erroneously apply to the background.
Datasets: The FairVE Benchmark
The authors introduced FairVE, the first dataset specifically designed to evaluate fairness in video editing. It consists of 640 video-text pairs covering 8 representative professions (CEO, doctor, technician, pilot, teacher, nurse, fashion designer, and librarian) across two genders. Videos were curated from the DAVIS benchmark and high-quality web clips, filtered to ensure a single human subject for clear evaluation.
Key Results
FAME was rigorously tested against state-of-the-art baselines like TokenFlow and VideoGrain, yielding the following insights:
- Fairness Alignment: FAME achieved significantly higher "Correction Ratios." In human evaluations, its Bias Correction scores frequently exceeded 80/100.
- Temporal Consistency: FAME achieved the lowest Warp-Err (1.66), a metric for video jitter. Baselines often exhibited errors nearly double this value (up to 3.08).
- Semantic Fidelity: FAME outperformed all baselines in CLIP-F (frame consistency) and TIFA (text-to-image alignment) scores, proving that fairness does not require sacrificing video quality.
Limitations and Future Work
While FAME represents a significant step forward, it is currently optimized for videos with a single human subject. Handling complex scenes with multiple actors or highly dynamic backgrounds remains an open challenge. Furthermore, while this study focuses on gender-profession stereotypes, the authors acknowledge the need to expand this framework to address racial, age, and intersectional biases in future research.
Summary updated