TL;DR
CogPortrait is a novel two-stage portrait animation framework that enables fine-grained control over eye-region dynamics (such as gaze direction, blinks, and eyebrow movements) directly from high-level text labels or instructions. By combining a hierarchical multi-agent LLM planning system with a Diffusion Transformer (DiT) video generator, it synthesizes highly realistic, nuanced facial expressions for complex states like thinking or drowsiness without requiring tedious motion-level driving inputs.
What Problem Does This Paper Address?
Current portrait animation methods suffer from a frustrating trade-off between user convenience and motion accuracy.
On one hand, label-driven methods let users input simple emotional categories (e.g., "angry") or macro-directives, but they lack the granularity to control subtle ocular dynamics like asymmetric blinks or gaze aversion. On the other hand, motion-driven methods use dense signals like facial keypoints or driving videos to achieve high fidelity, but they place a heavy input burden on the user. Furthermore, these methods struggle with "beyond-emotion" states (e.g., cognitive effort, drowsiness) and long-tail, irregular motion patterns (e.g., large-angle head rotations), often leading to over-smoothed, robotic movements or identity distortion.
CogPortrait bridges this gap by allowing users to achieve precise, fine-grained eye-region control using only high-level semantic labels or simple text instructions.
Methods
CogPortrait splits the animation task into two distinct stages: compiling semantic labels into physical keypoints, and then rendering those keypoints into a photorealistic video.
Stage 1: Hierarchical Agent Planning
Instead of directly mapping text to motion trajectories (which usually results in unnaturally smooth movements), CogPortrait uses three Chain-of-Thought (CoT) Multimodal Large Language Model (MLLM) agents to generate realistic keypoints:
- Planning Agent: Decomposes a high-level label (e.g., "cognitive effort") into a sequence of temporally structured events (e.g., "attentive onset" followed by "gaze shifts left with a mild squint").
- Composition Agent: Retrieves real behavioral snippets from a curated Prototype Library and stitches them together, scaling amplitudes to match the planned events while preserving natural, abrupt human dynamics.
- Critic Agent: Acts as a quality filter, enforcing semantic consistency and rule-based physiological constraints (e.g., verifying realistic blink durations and gaze-head coordination).
The final sequence is projected into 62 explicit eye-region keypoints using a FLAME mesh.
Stage 2: Video Generation with Keypoint Guidance
The keypoints, reference image, audio, and text prompt are fed into a Diffusion Transformer (DiT) backbone (based on Wan2.2). To maximize visual fidelity, the authors introduce two key enhancements:
- Dynamic Spatiotemporal CFG: Standard Classifier-Free Guidance (CFG) often causes global color shifts and washes out fine details. CogPortrait applies a temporal trapezoidal schedule (stronger guidance early on to lock in global structure) combined with a spatial Gaussian weighting map centered entirely on the eyes to boost local controllability.
- KTO-Based Refinement: To handle rare boundary cases like asymmetric eyebrow raises or extreme head turns, the model is fine-tuned using Kahneman-Tversky Optimization (KTO), aligning the model's outputs with human preferences using simple binary feedback.
Datasets
- Training Backbones: TalkVid and DH-FaceVid-1K.
- KTO Refinement: A curated preference dataset of 400 clips from DH-FaceDrasMvVid (for large head motions) and self-collected videos of asymmetric eyebrow movements and rapid blinking.
- Evaluation: Evaluated on the standard HDTF dataset.
- The EMH Benchmark: The authors introduce the new Emotion and Beyond-Emotion (EMH) benchmark. It spans 12 categories, including 6 core emotions (from MEAD) and 6 complex states (laughter, cognitive effort, low-arousal negative, social engagement, evasive response, drowsiness). It introduces two new Action Unit (AU) metrics: AU-F1 for activation correctness and AU-Temp for temporal trajectory fidelity.
Key Insights and Ideas
- Prototypes over Generation: Directly generating motion coordinates from text leads to "mushy," over-smoothed animations. Retrieving and compositing real human motion primitives preserves the micro-dynamics (like tiny eye twitches or rapid gaze shifts) that make a face look truly alive.
- Localized Guidance Matters: Treating an entire image uniformly during diffusion guidance hurts fine-grained control. Prioritizing the eye region spatially and structurally prevents the background from distorting while allowing the eyes to move sharply.
- KTO for Long-Tail Stability: Aligning video diffusion models on long-tail edge cases (like near-profile head turns) is traditionally difficult due to a lack of paired data. Utilizing KTO with per-sample binary feedback successfully mitigates identity drift during extreme poses.
Key Results
CogPortrait achieves state-of-the-art performance across both standard and fine-grained benchmarks:
- On HDTF: It achieves the best scores in visual quality (FID of 16.68, FVD of 32.90) and identity preservation (ID-Sim of 0.9214). Crucially, it reduces the eye-region landmark distance (Eye-LMD) by 38% compared to the previous best method, demonstrating vastly superior control accuracy.
- On EMH Benchmark: CogPortrait significantly outperforms audio- and label-driven baselines on temporal trajectory fidelity (AU-Temp), proving that its agent-based planning successfully translates high-level concepts into physically accurate expressions.
Limitations and Future Work
While CogPortrait excels at eye-region and head-pose control, mouth keypoints are currently omitted from the Stage 1 planning phase; mouth movements are instead generated implicitly via audio conditioning in Stage 2. A natural extension of this work would be to expand the hierarchical agent framework to holistically plan full-face dynamics, integrating lip-syncing, jaw movements, and micro-expressions into a single unified planning pipeline.
Summary updated