Metadata-Supervised Narration Alignment for Mining Computer-Use Screencasts

Computer-use agents that see only the screen are increasingly trained on screen recordings of people using software. Turning those recordings into training examples means answering two questions for every moment of video: what action just happened, and what was the person trying to do. Narrated tutorials answer the second question, but narrators rarely speak in step with their hands. They say "now I'll save this" three seconds before pressing Ctrl+S, or "that turned on the grid" after the fact. Current pipelines take the speech that falls in a fixed window around each action. That fixed window attaches intentions to the wrong actions, and it never registers keyboard actions that move no cursor.

We propose to learn this alignment rather than guess it. The training signal comes from the very structural metadata that pure-vision agents have abandoned. We will record a modest set of narrated sessions inside instrumented virtual machines, where input-event logs and accessibility trees reveal exactly when each action happened and which named element it touched. That record supervises a small cross-modal attention model whose input is pixels and speech only. The metadata is then discarded, and the model labels unlabeled YouTube screencasts. Metadata appears only during training, never at inference. The claim is that a model trained this way, which learns how narrators' timing is offset from their actions, produces cleaner mined trajectories than heuristic windows, and that cleaner trajectories measurably improve downstream agents.

Core Claim and Positioning

The closest work is VideoAgentTrek. It mined 1.52 million interaction steps from 10,000 hours of YouTube video using a cursor-based screen filter, an inverse-dynamics action extractor, and an LLM that writes an "inner monologue" from transcript text. Pretraining on these steps lifted OSWorld-Verified success from 9.3% to 15.8%. The authors acknowledge that narration leads or lags actions and that drags and key presses are hard. GUI-Narrator localizes action start and end frames with a cursor detector and captions cropped keyframes. It ignores audio entirely and names keyboard typing and hotkeys as its main failure. Screencast Tutorial Video Understanding fused detected Photoshop pop-ups and tool cues with video through attention, and it explicitly leaves out audio. These three papers motivate the proposal: each finds the gap and none closes it.

Our proposal differs in two ways. First, it learns narration–action correspondence with an explicit, asymmetric model of the time offset, instead of slicing by fixed windows. Second, it obtains ground truth for that correspondence from instrumented recordings, so the alignment is trained against real labels rather than LLM plausibility.

The structural-metadata literature shapes how we use metadata. UIBert and Multimodal Icon Annotation show that view hierarchies are noisy and lag the rendered screen. UFO and UINav document controls that expose nothing to accessibility APIs. Metadata therefore cannot be a dependable inference-time input, and the target domain of public videos has none anyway. These same flaws also limit how far we can trust metadata as a label source. For that reason, timestamps come from low-level input-event hooks, which are reliable. Accessibility trees supply only element names, and those names are used for matching to narration and checked by humans on the test split.

Research Program

The alignment model reads two streams. The first is visual: frames sampled at a few per second, encoded by a frozen pure-vision GUI encoder (we will use the Aguvis backbone). Each frame is paired with a high-resolution crop of the region that changed since the previous frame. Change regions are used rather than cursor boxes so that typing and shortcuts still produce a signal. The second stream is lexical: speech-recognizer word tokens with their timestamps. The model does three jobs. It detects action events at each frame, with classes that include click, type, hotkey, drag, and scroll. It reconstructs typed strings and hotkey identities. For each action it selects the narration span that expresses that action's intent, or declares that no span does.

For selection, each detected action $a$ at time $t_a$ queries narration word $w$ at time $\tau_w$ with the logit

$$ s_{a,w} = \frac{q_a^\top k_w}{\sqrt{d}} + b_\theta(\tau_w - t_a,\, c_w), $$

where $q_a$ and $k_w$ are the action query and word key in a shared $d$-dimensional space. The term $b_\theta$ is a learned bias over the signed offset. It is allowed to differ before and after the action, and it can depend on a discourse-cue embedding $c_w$, so that "now I'll" and "I just" pull in opposite temporal directions.

Two components are reused from earlier work. We apply Sparsemax over $s_{a,\cdot}$ so that each action attends to a compact span rather than a blur. We also add a learnable gate that mixes the retrieved context with the action's own visual features, as in the gated cross-attention of Kim et al., where it kept a large stream from washing out a small one. Here the gate lets the model output "no narration for this action," which matters because many actions go unnarrated. We also run attention in the reverse direction, with narration phrases querying frames. This follows the bidirectional design used to fuse asynchronous event-camera and frame streams in Object Tracking by Jointly Exploiting Frame and Event Domain. An utterance such as "hit control-shift-P" can then help locate a hotkey event whose visual trace is only a palette appearing.

Generative synchronization of the kind used by DDL-CXR is deliberately not used. That method synthesizes an updated image latent with diffusion to bridge stale chest X-rays and hourly vitals. It shows that explicitly modeling asynchrony helps, but its diffusion cost is unnecessary when the goal is to choose a span rather than to imagine a missing frame.

Supervision comes from about 300 hours of narrated sessions recorded by contractors. The sessions run in OSWorld-style Ubuntu virtual machines with LibreOffice, GIMP, VS Code, Chrome, and the terminal, and contractors are told to narrate as they would for a tutorial. Keyboard and mouse hooks give exact action times, types, and strings. Accessibility snapshots give the name of the target element. A narration span is labeled as the intent of an action when it mentions that element or its effect, using automatic matching followed by human verification on the evaluation split. Because public tutorials differ in style from paid recordings, we will also hand-annotate 500 YouTube clips as an out-of-domain test of alignment. Those clips have no metadata; annotators mark action times and intent spans by watching.

Baselines

The naive reference is the heuristic used by current miners. Cursor-based event detection is paired with the transcript inside a fixed window around each event, and an LLM writes the monologue, as in VideoAgentTrek. We will sweep window widths and asymmetric placements and tune them on the instrumented training split, so the baseline receives the same label budget as our model.

The strongest existing system for action detection and captioning is GUI-Narrator. It reports captioning gains of 13.7% to 41.3% on Act2Cap over untuned open models. There is no reported result on Lacuna for an absolute narration–action alignment score on any benchmark, so we will rerun GUI-Narrator ourselves on Act2Cap and on our splits.

For downstream agents, the reported numbers to beat come from VideoAgentTrek: 15.8% task success on OSWorld-Verified, against 9.3% for training on curated demonstrations alone, and 69.3% step accuracy on AgentNetBench, against 64.1%. Re-mining 10,000 hours is beyond our budget. We will therefore mine the same 1,000-hour subset with both pipelines, continue pretraining the same base model with equal steps, apply identical fine-tuning, and run three seeds.

Two ablations isolate what matters. Removing the offset bias $b_\theta$ yields plain cross-attention. Replacing Sparsemax and the gate with softmax tests whether the gating matters. Adding a third pipeline, our event detector paired with heuristic windows, separates the value of better detection from the value of better alignment.

Our model beats the heuristic if it raises intent-span F1 on the YouTube test clips by a clear margin over the best-tuned window and improves hotkey and typing detection recall over GUI-Narrator. The downstream test asks for at least three points of OSWorld-Verified success over the heuristic-mined model at matched data, consistent across seeds. OSWorld-Verified has only 369 tasks, so smaller differences will be reported as inconclusive.

Why not just…?

Why not just keep the accessibility tree and skip vision? The target data is public video, which carries no metadata. Aguvis also reports that its pure-vision agent uses about 70% fewer input tokens and costs 93% less than a DOM-fed GPT-4o on Mind2Web-Live. Our design already gives metadata its most defensible role, as a training-time labeler.

Why not just hand the whole clip to an omni-modal model and ask for the trajectory? This is a real competitor, and we will include it, prompting a strong audio-video model on the same clips. Diagnostic audio-visual benchmarks such as DAVE and Daily-Omni, as described in our source survey, suggest that end-to-end models trail explicitly aligned pipelines on temporal questions. No reported number for GUI screencasts was found on Lacuna, so our alignment F1 and detection recall comparison will settle the question directly.

Why not just use monotonic alignment, such as dynamic time warping between transcript and frame descriptions, plus text-difference recognition to catch typing? This is the most serious threat. Typing leaves visible text, so text-difference recognition may recover it without audio. We will build this baseline carefully and report results by action type. If it matches our model on typing and on intent F1, the remaining case for learned alignment rests on hotkeys and on non-monotonic narration, such as retrospective summaries, and we will report that narrower result honestly.

Decisive Checks

The first experiment determines whether the project is worth pursuing. On the instrumented recordings we will train agents three ways: with oracle intent spans from metadata, with heuristic-window spans, and with no monologue at all, then evaluate on AgentNetBench and OSWorld-Verified. If oracle alignment does not beat heuristic windows downstream, then narration timing is not the bottleneck and the proposal stops there.

If oracle alignment does help, the remaining checks follow in order. We measure how much of the oracle gap our metadata-free model closes on the YouTube test clips. We then report hotkey and typing recall against GUI-Narrator on Act2Cap. Finally, we run the matched 1,000-hour mining comparison.