Harvesting GUI Grounding Labels from Raw Screen Recordings via Cursor Kinematics

Agents that operate computers from screenshots alone, with no access to the hidden accessibility trees or web page structure, fail mostly at one step: pointing at the right pixel. On the desktop UI-Vision benchmark, a 32B model picks the correct action 85.9% of the time but grounds it correctly only 52.4% of the time (CUA-Suite). The best fix found so far is dense, human-verified data. GroundNext's 700K curated samples beat models trained on 9 million synthetic images (Grounding Computer Use Agents on Human Demonstrations). Producing that data, though, meant annotators marking keyframes and drawing millions of boxes. Meanwhile, a far larger supply of human demonstrations sits unused: screen recordings and tutorials, where every click is preceded by the cursor approaching a target, slowing down, holding still, and then causing a visible change right under it.

We propose to recover grounding supervision from these recordings automatically. Each recording is split into "actoms", short segments that each contain one interaction. The split uses two signals together: the cursor's motion (deceleration and stopping) and a localized visual change near where the cursor stopped. For every click actom, the system emits a (screenshot, target box, instruction) triple. A VLM writes the instruction from a crop of the target, the same way GroundCUA's own instructions were produced. The claim is that labels harvested this way, without OS event logs or human boxes, carry enough of the value of human-annotated grounding data that adding them improves a static grounding model on professional desktop benchmarks. The segmentation is a means to that end. It is judged by whether the labels it yields make a grounding model better.

What exists and what does not

The segmentation tools come from motion analysis of physical bodies. Kinematic holds, the pauses where hand velocity approaches zero, line up with cognitive boundaries (The Catchment Feature Model). Gesture streams can be over-segmented at velocity shifts and reversals (Unsupervised Analysis of Human Gestures). Split-and-merge without any training (SaM) and rate-reduction subspace clustering (TR²C) segment human activity video without labels. None of these has been applied to on-screen cursor trajectories. We reuse SaM's split-and-merge structure as the segmentation backbone, and SaM's known sensitivity to its window and merge thresholds is a constraint we must handle.

On the GUI side, three systems sit closest to this proposal. GUI-Narrator trains a supervised YOLOv8 cursor detector and a supervised boundary model in order to caption actions. It uses the cursor only as a crop anchor and reports that keyboard typing remains unresolved. Learning from Online Videos at Inference Time prompts a VLM to estimate action spans in tutorials and throws away "mere cursor movements" as noise. It feeds the resulting snippets to an agent in context and does not use them as training data for grounding. OS-Genesis synthesizes tasks from discrete before/after screenshot pairs that it collects itself, depends on accessibility trees to act, and never sees continuous video. Our proposal differs from all three in its output, which is training labels for a pixel-level grounding model, and in its segmentation signal, which comes from motion and is verified against the visual effect of the action. The CUA-Suite paper supplies the essential test bed: VideoCUA's 55 hours of 30 fps desktop video with synchronized cursor traces (CUA-Suite). It lets us hide the logs from the method and still score against them.

Core Claim

Click targets in screen video are revealed by a recognizable sequence: the cursor brakes, holds, and is followed by a change in the pixels near the hold point. Detecting this sequence across both signals at once makes an unsupervised extractor precise enough that its (screenshot, box, instruction) triples improve a grounding model about as much per sample as human-annotated GroundCUA data. Because the extractor needs only video, it can draw on recordings and applications that no annotation effort has covered.

Research Program

The system has four stages.

The first stage tracks the cursor from pixels. A small heatmap model learns to locate the cursor sprite, trained on VideoCUA frames using the logged traces as free labels. It is then applied to recordings that have no logs. We compare it against GUI-Narrator's style of detector and against plain sprite template matching.

The second stage proposes and confirms boundaries. For frame $t$, let $v_t$ be cursor speed, and let $\Delta_t$ be the visual change, measured as the feature distance between frames after and before $t$, inside a window $W(p_t)$ centered on the cursor position $p_t$. The candidate boundary score is

$$s_t = \exp(-v_t/\sigma_v)\cdot \Delta_t\big(W(p_t)\big) \;-\; \beta\,\Delta_t\big(\overline{W(p_t)}\big),$$

where $\sigma_v$ sets the speed scale, $\overline{W}$ is the screen outside the window, and $\beta$ penalizes changes that happen everywhere at once, such as scrolling or full-page loads. These are scored separately. The first factor implements the "hold" cue from gesture research. The second ties the boundary to a local effect, so blinking clocks and spinners far from the cursor do not fire. Keyboard actions have no cursor motion. For these, the same local-change term is centered on the text caret, located by the same visual model, instead of on the pointer. This targets the failure GUI-Narrator reports. SaM-style merging then fuses adjacent proposals whose local features stay in the same subspace. Window and merge thresholds are tuned on a small held-out set of VideoCUA applications and frozen before any other evaluation.

The third stage extracts labels. For each click actom, the target box is the connected region of local change that contains the hold point. If there is no visible change, the box comes from the hold point snapped to the boundaries of nearby visual elements. Clicks whose change region is ambiguous are dropped. A VLM writes direct, functional, and spatial instructions from the crop, following GroundCUA's recipe.

The fourth stage trains the grounding model. A Qwen2.5-VL-7B model is fine-tuned GroundNext-style on harvested data alone, on human data alone, and on both.

Baselines

The naive reference point is the base Qwen2.5-VL-7B model before grounding fine-tuning, together with a harvester that takes frames at uniform 1 fps and boxes the largest frame-difference region. This shows what video yields without kinematics.

The strongest existing method on the downstream question is GroundNext trained on human-verified GroundCUA. GroundNext-7B (RL) reports 70.6% average accuracy across ScreenSpot-Pro, OSWorld-G, MMBench-GUI, ScreenSpot-v2, and UI-Vision (Grounding Computer Use Agents on Human Demonstrations). On ScreenSpot-Pro alone, GUI-Actor-7B reports 44.6 (GUI-Actor). We retrain the GroundNext recipe ourselves at matched sample counts, backbone, optimizer steps, and RL budget. This removes differences in implementation. The decisive comparison is per-sample. Beating it means harvested labels reach at least 85% of the gain that an equal number of human-labeled samples gives over the base model. Separately, human data plus harvested data from applications outside GroundCUA should exceed human data alone by a margin larger than seed variance across three runs.

The strongest competitor on the segmentation step is VLM prompting as in Learning from Online Videos. That paper reports only downstream agent gains (3.5% on OSWorld and 4.2% on WebArena for tasks with tutorials) and no boundary accuracy; we found no reported boundary-accuracy result on Lacuna for it. We run it on the same VideoCUA clips with a fixed token budget, and score both its boundaries and its downstream labels. GUI-Narrator's supervised pipeline is a second segmentation baseline, trained on Act2Cap as published.

Two ablations isolate the key ingredient, the joint test of motion and local visual effect. One uses kinematics only (pause detection on the cursor trace). The other uses visual change only (SaM on full-frame features). The proposal wins only if the joint version beats both on precision of click localization and on the downstream grounding gain.

Why not just…?

Why not just use OS event logs? When logs exist they are better ground truth, and that is exactly how we use them: VideoCUA's traces score the method. The target regime is the much larger body of recordings with no logs, such as tutorials, remote-desktop sessions, and archived screencasts, where Learning from Online Videos had to fall back on pixels. Logs also give a click coordinate but no element extent. Our box-extraction step adds that extent, and its accuracy is measured directly against GroundCUA boxes.

Why not just prompt a frontier VLM to slice the video? It is a real competitor, and it is run head to head. Against it stand the cost (GUI-Narrator's cursor crops cut visual tokens to about 0.38× of the original while improving captions) and coarse timing that blurs the hold moment. If the VLM slicer matches our boundary F1 and yields equally useful labels, the contribution collapses to a cheaper approximation of it, and we would report that.

Why not just detect cursor pauses? This is the most serious threat, because pauses alone may find most clicks. Pauses also fire on reading and hesitating, however, and they say nothing about the target's extent or about typing. The kinematics-only ablation settles the question. If it matches the joint method on downstream gain, the visual-effect term is unnecessary and the paper becomes a study of how cheaply video can be mined.

Why not just generate more synthetic grounding data? GroundNext already showed that 700K curated human samples beat 9M synthetic ones. The question here is whether human behavior in video can stand in for human annotation, and the matched-count comparison answers it.

Proposed Experiments

Segmentation is scored on held-out VideoCUA applications, with logged clicks and keystrokes as ground truth. Metrics are boundary F1 at 0.5 s and 1 s tolerance, click-point error in pixels, and box IoU against GroundCUA. Results are broken down by action type, with typing and drags reported separately. We also report temporal F1 and Mean over Frames on Act2Cap for comparison with GUI-Narrator. For calibration against physical-motion segmentation, SaM reaches 55.9% F1 on Breakfast, and we expect GUI streams to be easier on boundaries but harder on clutter. Robustness is tested by injecting distractor animations (clocks, video panes) and measuring the drop in false-positive boundaries relative to the visual-only ablation.

Downstream, grounding models are evaluated on ScreenSpot-Pro, ScreenSpot-v2, OSWorld-G, MMBench-GUI, and UI-Vision. UI-Vision's spatial-instruction subset gets particular attention, since GroundNext falls from about 70% to about 50% on it. Harvested data includes the approach direction and nearby elements, which may supply better spatial-relation instructions. Two checks decide the outcome. The first is the matched-count harvested-versus-human comparison on applications GroundCUA covers. The second is the gain from harvesting applications it does not cover, especially canvas-heavy tools like Darktable, where UI-Vision reports 3.6% success at 50 px. A final agent-level test pairs the best grounding model with a fixed planner on OSWorld-Verified. GroundNext-3B with o3 reports 50.6% there, and the planner is held constant so that any difference reflects grounding alone.