People who find face-to-face interaction hard, such as many autistic learners and people preparing for job interviews, benefit from practicing in "low-stakes" simulations. These are virtual settings where a misread cue costs nothing, and where the same moment can be replayed, slowed down, or simplified. Today's simulations are built in game engines with hand-animated avatars. They are repeatable, but they look rigid and cannot show the small signals that matter in real conversation: a half-second glance away, a delayed nod, a brow that tightens. Modern video generators produce realistic people, but they offer no control over any single signal. If you ask for "the listener looks away," the whole scene changes: identity, lighting, and timing all shift along with the gaze.
This note proposes a video generator built in two levels, with a small set of named cue channels in between. The top level writes a timed track of those channels (gaze direction, head pose, nods, brow and mouth movement, response latency). The bottom level is a video renderer trained so that changing one channel changes only that signal in the output. This makes it possible to produce minimal pairs: two videos that are identical except for one cue. The same design gives a dial for how strong that cue is. Minimal pairs and graded cue strength are exactly what structured rehearsal needs, from "simplify the cue" early on to "make it realistic and subtle" later. Neither current single-pass video generators nor game-engine sandboxes can provide both realism and this kind of isolation.
Core Claim
The claim is that the intermediate layer of a hierarchical video generator can serve as a control surface for behavioral experiments, not only as a way to save compute. If the renderer is trained with an explicit penalty for spillover between cue channels, editing one channel should produce three things that prompt editing of end-to-end models does not. First, the edited cue appears in the pixels as specified. Second, the other channels, the person's identity, and the background stay fixed. Third, human viewers' ability to detect the cue rises steadily as its strength is turned up. The third property is the one a rehearsal curriculum depends on, and the proposal treats it as the decisive test.
The claim is deliberately narrower than "hierarchical social video generation." Adding a social planning stage above a video cascade is the obvious move at this intersection. On its own it would be hard to tell apart from simply having a language model write a better prompt. What is specific here is cue isolation, together with a way to measure it.
Where This Sits
The rehearsal literature motivates the target property. Reviews of virtual-reality interventions for neurodevelopmental conditions stress uniform, repeatable scenarios that can be paused and replayed (Capobianco et al.). Work on virtual reality in early education argues for deliberately filtering and pacing cues before exposing learners to noisy real interactions (Hou). The closest deployed system, AIRoad (Cao et al.), uses GPT-4 Turbo to drive Unity-rendered drivers for autistic children learning street-crossing cues. It reports a usability (SUS) score of 85.71 against 80.36 for video tutorials. It also reports two limits that this proposal inherits as constraints: a 6% rate of behavioral errors from the language model, and cartoonish assets that lack natural gestures. AIRoad already pre-generated vehicle states to hide latency. That precedent means an offline library of rendered minimal pairs is an acceptable first deployment form. Real-time generation is not a prerequisite.
Hierarchical video synthesis supplies the method, and it also warns about a weakness. Cascaded generators such as CVG (Castrejón et al.), Show-1 (Zhang et al.), LaVie (LAVIE), and LUVE (Zhao et al.) all split generation along resolution or frequency. They fix coarse motion first and add detail later. CVG notes that coarse-stage mistakes are then locked in and sharpened downstream. This proposal turns that weakness into an advantage: because the cue track is explicit and readable, it can be checked before any pixels are rendered. Language-model plan errors of the kind AIRoad reports can therefore be caught cheaply at the structural level.
Two intermediate representations come closest to this one. Relational layouts (Bodla et al.) predict 2D pose keypoints and object boxes and then render them. On the Bimanual dataset this halved FID from 69.92 to 35.82. However, the layouts cover only human-object manipulation and carry no faces or gaze. DIM (Tran et al.) models both partners in a conversation using discrete 3D facial motion codes pretrained on 850 hours of CANDOR. It reaches a facial-expression Fréchet distance of 23.88 on ViCo. DIM stops at mesh parameters, and its authors concede that rendering through PIRenderer distorts faces and needs per-person fine-tuning. This proposal reuses DIM's code space as the top level. It adds what DIM lacks: a photorealistic renderer that preserves the cue track, and an isolation guarantee.
Evaluation work defines the gap and sets the evaluation rules. SVBench (Peng et al.) shows that end-to-end generators handle social cues poorly. Scores range from 27.6% (LTX-1.0) and 30.8% (HunyuanVideo) to 79.6% (Sora2pro), with "literalism" failures. SVBench also finds that human judges are stricter about social-logic errors than its Gemini judge. For that reason, humans, not a vision-language model, are the primary judge here. VBench (Huang et al.) measures visual quality. It serves as a guardrail and not as the target, since SVBench shows that visual quality and social coherence come apart.
Research Program
The domain is face-to-face listening in a two-person conversation. The learner watches a speaker addressing a listener and must judge whether the listener is engaged, confused, or disengaged. This matches the emotion-recognition and peer-interaction goals in the rehearsal literature. It also has the data needed: CANDOR for training and ViCo for comparison with DIM.
The top level produces a cue track $c = (c_1, \dots, c_K)$, a set of $K$ time-aligned channels. The channels are head rotation, gaze direction, blinks, brow action, mouth and smile action, discrete nod events, and response latency relative to the speaker's pauses. Facial channels come from DIM's quantized FLAME codes. Gaze and head pose come from an off-the-shelf estimator run on CANDOR. The track can be produced three ways: sampled from a DIM-style model conditioned on the speaker, written by a language model from a scenario description, or edited by hand by a clinician. In every case a rule-based checker validates it before rendering. The checker verifies, for example, that "disengaged" contains no sustained mutual gaze and that timings are physically possible.
The bottom level is a latent video diffusion renderer, fine-tuned from an open text-to-video backbone. A diffusion model generates by gradually denoising random noise into a sample. The renderer is conditioned on a reference image of the listener, the speaker's audio, and the cue track, which is supplied as per-frame embeddings and a sparse keypoint map. Following LUVE's logic of fixing motion first, the renderer produces low-resolution motion, and a separate stage adds high-frequency facial detail without touching timing.
The key ingredient is a paired isolation objective used during fine-tuning. Let $R(c, z)$ be the rendered video for cue track $c$ and shared noise $z$. Let $c'$ equal $c$ except in channel $k$. Let $E_j$ be the tracker that re-extracts channel $j$ from video, and let $d_{\mathrm{id}}$ be a distance between face-identity and background embeddings. We add
$$ \mathcal{L}_{\mathrm{iso}} = \big\| E_k(R(c', z)) - c'_k \big\| + \lambda \Big( \sum_{j \neq k} \big\| E_j(R(c', z)) - E_j(R(c, z)) \big\| + d_{\mathrm{id}}\big(R(c', z), R(c, z)\big) \Big), $$
where $\lambda$ weights the spillover penalty. The first term asks that the edited cue actually appear. The rest asks that nothing else move. At evaluation time the same two quantities are reported separately, as cue fidelity and leakage.
The cue-strength dial scales an edited channel's deviation from a neutral track by a factor $\alpha \in [0, 1]$. Examples are the depth of a nod or the angle of a gaze aversion. For a curriculum, what matters is that the proportion of viewers who detect the cue rises smoothly with $\alpha$, following an ordinary S-shaped psychometric curve. It should not be flat or jump abruptly.
Baselines
The naive reference point is branching real footage, which is the video-library approach. For each target cue, retrieve the nearest CANDOR listener clip that shows it, matched on speaker segment where possible. This is maximally realistic. By construction, though, its "pairs" differ in identity, lighting, and timing, so it sets the ceiling for realism and the floor for isolation. No generation budget applies to it.
The strongest competitor for realism combined with editability is a language-model planner feeding a single-pass open generator. A language model rewrites each scenario into an explicit physical shot description, in the style of SVBench's prompt surgery. HunyuanVideo, fine-tuned on the same CANDOR subset with the same reference-image conditioning, then renders it. Minimal pairs are made by editing only the cue sentence and fixing the random seed. It is the strongest competitor because it is what practitioners would try first, and because larger proprietary models appear to handle social content better. The strongest reported number on social content is Sora2pro's 79.6% SVBench overall accuracy (SVBench). Sora2pro and Veo-3.1 will be run as prompt-only references if access allows, without fine-tuning, and reported separately. No reported result was found on Lacuna for cue isolation or leakage for any video generator. Beating this baseline means matching it within a pre-set margin on VBench's subject-consistency and imaging-quality dimensions, while halving leakage and showing a monotone psychometric curve where the baseline's curve is flat or non-monotone.
The strongest existing structured method is DIM plus PIRenderer, which uses the same cue codes rendered by DIM's own pipeline (Tran et al.). DIM's reported 23.88 facial Fréchet distance on ViCo is the reference for the top level. We will report our re-extracted Fréchet distance on ViCo under the same protocol. Beating DIM means equal cue fidelity with better human-rated realism and identity stability, without per-person fine-tuning.
We also include an engine-plus-restyle baseline. A rigged avatar is driven by the same cue track and rendered in a game engine. It is then restyled to photorealism by applying the same diffusion backbone video-to-video with reference-image conditioning. No paper in the file reports this pipeline, so no number exists for it. It gets the same fine-tuning data and steps.
Two ablations isolate the proposal's ingredients. The first removes $\mathcal{L}_{\mathrm{iso}}$ and keeps the cue-track conditioning, which tests whether conditioning alone gives isolation. The second replaces the cue track with its text description on the same renderer, which tests whether the structured intermediate layer matters at all.
All learned systems get the same data, the same GPU-hour budget, and the same number of hyperparameter trials. Pairs are generated under matched seeds wherever the method allows.
Why not just…?
Why not just let a language model write precise prompts for a strong video generator? This is the main threat, and the proposal is designed so the experiments settle it rather than argue around it. SVBench shows that even well-surgered physical prompts yield literalism and identity drift in open models, with HunyuanVideo scoring 30.8%. But SVBench never tests fixed-seed minimal-pair editing, which is the actual question here. If the planner-plus-generator baseline reaches comparable leakage and a monotone detection curve, the conclusion is that the structured layer is unnecessary, and we will report that. The text-track ablation on our own renderer separates "better model" from "better control surface."
Why not just keep using game-engine avatars, which are already isolated and real-time? Game engines give perfect isolation for free, and AIRoad shows they are usable. However, the file's own account of their limits is that their pre-baked animations lack the micro-expressions and natural gestures that matter for recognizing social cues. The engine-plus-restyle baseline is the fair version of this alternative. If restyling preserves subtle cues at low $\alpha$, the cue track should simply be fed from the engine, and our contribution narrows to the isolation loss and evaluation protocol. We expect restyling to struggle exactly at low cue strength, where subtle hand-animated motion is washed out. The low-$\alpha$ end of the psychometric curve is the test.
Why not just use DIM with a better face-reenactment network? DIM already has the right top level, and its authors name the renderer as the weak link. A stronger reenactment model would, however, still animate only the face over a fixed image. It would not handle audio-synchronized upper-body movement, gaze relative to the speaker, or response latency, and it provides no isolation guarantee. The DIM-plus-PIRenderer baseline and the no-isolation-loss ablation together show how much of any gain comes from rendering quality and how much from the paired objective.
Why not just use a library of real clips with branching? AIRoad found adaptive simulation preferred over video tutorials (SUS 85.71 versus 80.36), although that comparison confounds adaptivity with the medium. The deeper problem is that real clips cannot form minimal pairs or graded strength ladders. The retrieval baseline measures this directly as leakage.
Proposed Experiments
The first experiment tests isolation and fidelity. On 500 held-out CANDOR speaker segments, we edit each of the $K$ channels in both directions and render pairs under every system. We report cue fidelity, leakage, and identity distance using trackers that were not used in training. For example, the gaze estimator used for evaluation differs from the training one, so that no system can overfit the measurement.
The second experiment is the decisive human check. Adult viewers see pairs and single clips at five cue strengths, and each viewer judges whether a named cue is present and what the listener's state is. We fit a psychometric curve per system and channel. The proposal succeeds only if our system's curves are monotone, and if the realism ratings of its high-$\alpha$ clips are statistically indistinguishable from the retrieval baseline. Because SVBench found humans stricter than its VLM judge, a Gemini-style judge is reported only as a secondary score, along with its agreement with humans.
The third experiment is a guardrail. VBench quality dimensions on all generated clips confirm that isolation was not bought with blur. ViCo Fréchet distance confirms that the top level matches DIM.
The fourth experiment is a plan-checking test. We inject language-model-written cue tracks and measure how many errors the rule-based checker catches before rendering. This is compared against AIRoad's 6% behavioral error rate as the only available reference point.
A clinical pilot with autistic adolescents, comparing a graded minimal-pair curriculum with an ungraded one, is planned only after the second experiment succeeds. Given the small cohorts typical in this literature, it will be framed as feasibility and usability, not efficacy.
References
- Current virtual reality-based rehabilitation interventions in neuro-developmental disorders at developmental ages
- The Potential and Challenges of Virtual Reality Technology in Early Childhood Education
- Designing LLM-simulated Immersive Spaces to Enhance Autistic Children's Social Affordances Understanding in Traffic Settings
- Cascaded Video Generation for Videos In-the-Wild
- Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation
- LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models
- LUVE : Latent-Cascaded Ultra-High-Resolution Video Generation with Dual Frequency Experts
- Hierarchical Video Prediction Using Relational Layouts for Human-Object Interactions
- DIM: Dyadic Interaction Modeling for Social Behavior Generation
- SVBench: Evaluation of Video Generation Models on Social Reasoning
- VBench: Comprehensive Benchmark Suite for Video Generative Models