CogPortrait: Fine-Grained Eye-Region Control in Portrait Animation via Hierarchical Agent Planning

He Feng, Yongjia Ma, Donglin Di, Lei Fan, Tonghua SuarXiv 2026 · arXiv

TL;DR

CogPortrait is a novel two-stage portrait animation framework that enables fine-grained control over eye-region dynamics (such as gaze direction, blinks, and eyebrow movements) directly from high-level text labels or instructions. By combining a hierarchical multi-agent LLM planning system with a Diffusion Transformer (DiT) video generator, it synthesizes highly realistic, nuanced facial expressions for complex states like thinking or drowsiness without requiring tedious motion-level driving inputs.

Figure 1: Comparison with prior portrait animation paradigms, quantitative results on the EMH benchmark, and OOD cases generated by CogPortrait. Left: comparison of prior label-driven [26, 45] and motion-driven methods [18, 60] with the proposed CogPortrait framework, highlighting direct fine-grained eye-region control from high-level labels. Central: radar-chart comparison with baselines [8, 22, 24, 26, 35, 45, 50, 60] on the EMH benchmark, where larger areas indicate better performance. Right: OOD cases generated by CogPortrait, including fine-grained instruction, laughter, cognitive effort, drowsiness, large- angle head movement, and control of gaze and eyebrow motion.

What Problem Does This Paper Address?

Current portrait animation methods suffer from a frustrating trade-off between user convenience and motion accuracy.

On one hand, label-driven methods let users input simple emotional categories (e.g., "angry") or macro-directives, but they lack the granularity to control subtle ocular dynamics like asymmetric blinks or gaze aversion. On the other hand, motion-driven methods use dense signals like facial keypoints or driving videos to achieve high fidelity, but they place a heavy input burden on the user. Furthermore, these methods struggle with "beyond-emotion" states (e.g., cognitive effort, drowsiness) and long-tail, irregular motion patterns (e.g., large-angle head rotations), often leading to over-smoothed, robotic movements or identity distortion.

CogPortrait bridges this gap by allowing users to achieve precise, fine-grained eye-region control using only high-level semantic labels or simple text instructions.

Methods

CogPortrait splits the animation task into two distinct stages: compiling semantic labels into physical keypoints, and then rendering those keypoints into a photorealistic video.

Figure 2: Overview of the proposed CogPortrait. Stage 1 compiles a high-level label, a reference portrait, and optional fine- grained instructions into facial keypoints through four layers: Event Planning produces staged events, a CoT composition agent performs prototype retrieval to construct a control sequence, a critic agent applies semantic and physiological checks to refine the sequence, and a mapping layer projects the refined controls into facial keypoints. The critic can send revision signals back to the composition or planning layer. Stage 2 generates the final portrait animation from the reference portrait, driving audio, a text prompt, and the facial keypoints via a DiT-based generation backbone with dynamic CFG with eye-region-aware reweighting.

Stage 1: Hierarchical Agent Planning

Instead of directly mapping text to motion trajectories (which usually results in unnaturally smooth movements), CogPortrait uses three Chain-of-Thought (CoT) Multimodal Large Language Model (MLLM) agents to generate realistic keypoints:

  1. Planning Agent: Decomposes a high-level label (e.g., "cognitive effort") into a sequence of temporally structured events (e.g., "attentive onset" followed by "gaze shifts left with a mild squint").
  2. Composition Agent: Retrieves real behavioral snippets from a curated Prototype Library and stitches them together, scaling amplitudes to match the planned events while preserving natural, abrupt human dynamics.
  3. Critic Agent: Acts as a quality filter, enforcing semantic consistency and rule-based physiological constraints (e.g., verifying realistic blink durations and gaze-head coordination).

The final sequence is projected into 62 explicit eye-region keypoints using a FLAME mesh.

Stage 2: Video Generation with Keypoint Guidance

The keypoints, reference image, audio, and text prompt are fed into a Diffusion Transformer (DiT) backbone (based on Wan2.2). To maximize visual fidelity, the authors introduce two key enhancements:

  • Dynamic Spatiotemporal CFG: Standard Classifier-Free Guidance (CFG) often causes global color shifts and washes out fine details. CogPortrait applies a temporal trapezoidal schedule (stronger guidance early on to lock in global structure) combined with a spatial Gaussian weighting map centered entirely on the eyes to boost local controllability.
  • KTO-Based Refinement: To handle rare boundary cases like asymmetric eyebrow raises or extreme head turns, the model is fine-tuned using Kahneman-Tversky Optimization (KTO), aligning the model's outputs with human preferences using simple binary feedback.

Datasets

  • Training Backbones: TalkVid and DH-FaceVid-1K.
  • KTO Refinement: A curated preference dataset of 400 clips from DH-FaceDrasMvVid (for large head motions) and self-collected videos of asymmetric eyebrow movements and rapid blinking.
  • Evaluation: Evaluated on the standard HDTF dataset.
  • The EMH Benchmark: The authors introduce the new Emotion and Beyond-Emotion (EMH) benchmark. It spans 12 categories, including 6 core emotions (from MEAD) and 6 complex states (laughter, cognitive effort, low-arousal negative, social engagement, evasive response, drowsiness). It introduces two new Action Unit (AU) metrics: AU-F1 for activation correctness and AU-Temp for temporal trajectory fidelity.
Overview of the newly introduced EMH benchmark, detailing video durations, face resolutions, and sample categories.

Key Insights and Ideas

  • Prototypes over Generation: Directly generating motion coordinates from text leads to "mushy," over-smoothed animations. Retrieving and compositing real human motion primitives preserves the micro-dynamics (like tiny eye twitches or rapid gaze shifts) that make a face look truly alive.
  • Localized Guidance Matters: Treating an entire image uniformly during diffusion guidance hurts fine-grained control. Prioritizing the eye region spatially and structurally prevents the background from distorting while allowing the eyes to move sharply.
  • KTO for Long-Tail Stability: Aligning video diffusion models on long-tail edge cases (like near-profile head turns) is traditionally difficult due to a lack of paired data. Utilizing KTO with per-sample binary feedback successfully mitigates identity drift during extreme poses.

Key Results

CogPortrait achieves state-of-the-art performance across both standard and fine-grained benchmarks:

  • On HDTF: It achieves the best scores in visual quality (FID of 16.68, FVD of 32.90) and identity preservation (ID-Sim of 0.9214). Crucially, it reduces the eye-region landmark distance (Eye-LMD) by 38% compared to the previous best method, demonstrating vastly superior control accuracy.
  • On EMH Benchmark: CogPortrait significantly outperforms audio- and label-driven baselines on temporal trajectory fidelity (AU-Temp), proving that its agent-based planning successfully translates high-level concepts into physically accurate expressions.
Qualitative comparison on HDTF and the EMH benchmark, highlighting CogPortrait's precise eyebrow and eye control compared to baselines.

Limitations and Future Work

While CogPortrait excels at eye-region and head-pose control, mouth keypoints are currently omitted from the Stage 1 planning phase; mouth movements are instead generated implicitly via audio conditioning in Stage 2. A natural extension of this work would be to expand the hierarchical agent framework to holistically plan full-face dynamics, integrating lip-syncing, jaw movements, and micro-expressions into a single unified planning pipeline.

Summary updated

Abstract

Portrait animation methods have achieved substantial visual quality and lip synchronization, but fine-grained manipulation of the eye region still faces a trade-off between input granularity and motion accuracy. Existing methods using emotion labels or coarse text prompts are insufficient for describing subtle ocular dynamics, whereas approaches based on Action Units or driving videos provide higher fidelity at the cost of a heavier input burden. These limitations are still restrictive for beyond-emotion states (e.g., thinking) and drowsiness. In light of the above, we propose CogPortrait, a two-stage framework that generates portrait animations from high-level labels. In the first stage, three chain-of-thought Multimodal Large Language Models (MLLMs) agents compile high-level labels into facial keypoints through temporal event planning, prototype retrieval, and composition from a real-behavior library, and semantic-physiological constraint enforcement. In the second stage, a DiT-based video generation backbone synthesizes the final animation conditioned on the keypoints, reference portrait, audio, and text prompt, enhanced by a dynamic classifier-free guidance strategy with eye-region-aware reweighting and KTO-based refinement for boundary cases. We further introduce the EMH benchmark covering diverse emotions and beyond-emotion categories with two AU-level metrics for evaluating fine-grained eye-region and head-motion control. Extensive experiments on HDTF and the EMH benchmark demonstrate that CogPortrait achieves more precise eye-region control than existing methods while maintaining supe- rior visual quality and identity consistency
BibTeX
@misc{feng2026cogportrait,
  author = {He Feng and Yongjia Ma and Donglin Di and Lei Fan and Tonghua Su},
  title = {{CogPortrait: Fine-Grained Eye-Region Control in Portrait Animation via Hierarchical Agent Planning}},
  year = {2026},
  eprint = {2605.28056},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.28056},
}
Figure