Showing Learners Their Own Corrected Motion, Only As Much As They Need

When a coach says "your elbow flares out on the release," a novice often cannot turn that sentence into a different movement. Watching an expert helps somewhat, but the expert has a different body, stands somewhere else, and differs in dozens of ways that have nothing to do with the critique. This proposal builds a video generator that takes a learner's recorded attempt and a coach's time-stamped critique, and produces a video of the learner themselves, in their own clothes and setting, performing the movement with only the criticized part corrected. Body, room, and the uncriticized parts of the motion stay fixed. Only the faulty segment of the movement changes.

This is possible because of a design idea from video generation called content-motion decoupling: the model first plans motion in a compact form, such as a 3D skeleton trajectory, and only then paints appearance over it. Because motion and appearance are separate, the system can also decide how much of the correction to reveal. It can highlight only the error window, show a ghost skeleton, show the corrected self-video for the error window, or show the full corrected performance. The second half of the proposal borrows from facilitator-led teaching, where the instructor gives the smallest hint that lets the learner succeed rather than a full answer. We estimate, per learner and per critique, the least revealing level that still works. Our main claim is that a minimal, self-referenced, motion-only correction, given at a level matched to the learner's skill, produces larger practice gains than text critiques, expert videos, or always-full corrections. Decoupling is what makes graded scaffolding of this kind something one can build at all.

What exists and what is missing

The ingredients exist separately. Ego-Exo4D records skilled activity from first- and third-person cameras, with 3D body and hand pose ground truth and time-aligned expert commentary that critiques technique. It uses these only for diagnostic tasks such as proficiency estimation. It does not generate feedback, and this proposal turns its commentary into a conditioning signal. EgoExo-Gen predicts hand-object interaction masks in a first-person view from a third-person demonstration, then renders video conditioned on those masks. It faithfully copies whatever the demonstration contains and never edits motion toward a stated goal. We reuse its motion-first structure but replace "copy the demo" with "edit the learner's own trajectory as the critique asks."

Two further lines of work supply method. MIMIC and Motion-I2V plan motion first and render texture second. MIMIC also documents two failure modes that constrain our design: a moving mask is ambiguous between object motion and camera motion, and errors cascade from the first stage. We therefore use 3D skeletons from static exocentric cameras rather than 2D masks. Keyframe-Guided Creative Video Inpainting and VideoStudio show that propagating from reference frames preserves appearance better than editing the whole video volume, which motivates rendering from the learner's own frames.

On the instruction side, Cui & Sachan formalize a zone of proximal development (the band of tasks a learner fails alone but solves with help) for language models. They fit an item-response-theory model, a statistical model of item difficulty and learner skill, to decide when to give in-context examples. They show that unnecessary examples hurt accuracy on tasks already within reach. KSC-PaL finds that student learning tracks shifts in who takes initiative ($R^2 = 0.20$) rather than how much the tutor says. Both results motivate withholding full corrections, but both live in text or symbolic domains.

ReaDe is the closest instruction-to-video system. It expands short user instructions into dense captions through reasoning plus reinforcement learning. It works purely at the prompt level, always adds detail, and does not model the user. That runs against the evidence above that more guidance is not always better.

Core Claim

Holding appearance fixed and editing only the criticized slice of a learner's motion yields corrective videos that are more faithful to the critique and less confounded than expert footage. Because the edit lives in an explicit motion representation, the amount of correction shown becomes a controllable variable. A learner-specific estimate of the least sufficient level will beat both the weakest feedback (text only) and the strongest (full corrected video) on real practice gains, especially for more skilled learners.

Research Program

The system has three parts.

The motion editor is a diffusion model over 3D body-and-hand pose sequences. It is conditioned on three inputs: the learner's own trajectory, the critique text, and the critique's time stamp. It is trained to produce a trajectory that (a) matches expert executions of the same task in the critiqued window and (b) matches the learner's original trajectory elsewhere. Training pairs come from Ego-Exo4D novice attempts with commentary, with targets drawn from expert takes of the same task and temporally aligned. There is no ground-truth "corrected self," so supervision is distributional within the window and reconstructive outside it.

The renderer is a pose-conditioned image-to-video diffusion model initialized from the learner's own exocentric frames. It re-synthesizes only frames in and around the edited window and blends them back in.

The scaffold selector extends Cui & Sachan's item-response approach. Critiques are embedded as items, learners get a latent skill vector initialized from an Ego-Exo4D-trained proficiency estimator, and the model predicts which scaffold level is the lowest at which the next attempt will fix the error. The four levels are: error window only; window plus ghost skeleton; corrected self-video of the window; full corrected self-video. The selector is updated online from each attempt's outcome.

The work focuses on Ego-Exo4D's basketball and soccer drills. Their critiques are mostly about whole-body kinematics, they have full 3D pose, and they can be practiced in a short lab session.

Baselines

The naive reference point is practice with no feedback beyond repeated attempts. It sets the floor that any intervention must clear in the human study.

The text-only baseline gives the learner the coach's critique as written in Ego-Exo4D. It isolates whether any video adds value beyond the words the coach already produced.

The retrieved-expert baseline shows the closest expert take of the same drill from the same camera angle. It is the cheapest strong competitor and receives the same viewing time and replay controls as our condition.

The strongest existing generative method is an EgoExo-Gen-style pipeline that translates the expert demonstration into the learner's scene. It is retrained on the same basketball and soccer subset with the same compute budget and the same number of tuning runs as our system. On Ego-Exo4D cooking it reports SSIM 0.537 and FVD 1031.693 (EgoExo-Gen). It has no reported result on the physical-skill domains or on any critique-faithfulness or learning metric (no reported result found on Lacuna for EgoExo-Gen on Ego-Exo4D sports). We will therefore report its numbers on our split rather than borrow them.

The monolithic baseline is a single text-guided video editor given the learner's clip and the critique. It tests whether decoupling matters at all.

Two ablations isolate the key ingredients. The first renders our corrected trajectory on the expert's appearance instead of the learner's, which isolates "self" content. The second fixes every learner at the full-correction level, which isolates calibration.

Beating the baselines requires different evidence offline and in the human study. Offline, our edits must be scored as more expert-like by a held-out proficiency estimator and judged critique-faithful by blind coaches, while showing lower pose change outside the window and higher identity similarity than EgoExo-Gen-style translation and monolithic editing. In the human study, calibrated scaffolding must yield a statistically reliable larger gain in blind coach-rated form than both retrieved-expert video and fixed full correction.

Why not just…?

Why not just show the learner an expert video? This is the real threat. Expert footage is free, and if learners extract the relevant difference themselves, generation adds nothing. Our view is that an expert differs from the learner in body shape, stance, and many uncriticized details, so the critique-relevant difference is buried. That is an argument, not evidence. The retrieved-expert condition is in the human study precisely to settle it. If it matches our method, we will report that self-referenced generation is unnecessary for these drills.

Why not just give the coach's text critique, perhaps expanded by a language model? ReaDe shows that dense expansion helps a video generator, raising the VBench Multiple Objects score on CogVideoX-2B from 40.17 to 71.89 (ReaDe). A generator is not a novice, however, and converting words into kinematics is exactly the step novices fail at. The text-only condition measures this directly.

Why not just edit the video end to end with one model? Decoupled pipelines outperform single-stage baselines in related settings. EgoExo-Gen's zero-shot FVD on H2O was 1360.477 versus 1740.031 for the next-best baseline, and MIMIC was preferred 42.88% of the time against 27.8% for the next competitor. More importantly, a monolithic editor exposes no motion variable, so graded scaffolding cannot be expressed in it. The monolithic baseline tests the fidelity half of this claim.

Why not just always show the full correction? Cui & Sachan find that unneeded guidance degrades performance, and Kersey et al. tie learning to learner initiative. Neither result has been tested on motor skills. The fixed-full-correction ablation will show whether calibration matters here or whether more is simply better.

Proposed Experiments

Offline evaluation. On held-out Ego-Exo4D basketball and soccer takes with commentary, we measure four things: proficiency-estimator shift toward expert inside the window; pose deviation outside it; identity similarity and FVD of rendered frames; and blind coach ratings of whether the video enacts the critique.

Human study. A pre-registered study with novices randomized across the five feedback conditions practices a basketball shooting drill over three sessions. Outcomes are blind coach ratings and automatic proficiency scores on a retention test one week later and on a transfer drill not used in training.

Decisive checks. Two results determine the conclusion. The first is whether calibrated self-video beats retrieved expert video on retention; if not, the generative half of the proposal fails. The second is whether skill moderates the gap between calibrated and full correction, with skilled learners helped less or harmed by full correction. If this interaction appears, it is the first motor-skill evidence that the zone-of-proximal-development pattern seen in language models also holds for visual motion feedback.