Analyst Eye Movements as Free Labels for Adapting Tiny-Vehicle Detectors to New Regions

Detectors that find tiny moving vehicles in wide-area aerial video, where each car covers 5 to 20 pixels, are trained and tested on one or two cities. Nobody has measured how they hold up when flown over a different place with different roofs, roads, and building heights. This matters because it is exactly where they are used. In every new region a human analyst is already watching the footage, and when people follow a moving car with their eyes they produce a distinctive eye movement called smooth pursuit: the eye glides at the target's speed instead of jumping between fixed points. We propose to treat the analyst's eye movements, recorded while they do their normal job, as a cheap and noisy labelling signal. We turn pursuit episodes into point labels for "a moving vehicle is here, travelling at this velocity," and use these labels to adapt a spatio-temporal tiny-vehicle detector to the new region without anyone drawing a box.

The central technical idea is to match the gaze to candidate objects by velocity rather than by position. At normal viewing zoom, eye-tracker error of about a degree corresponds to tens of image pixels, which is larger than the vehicles themselves, so gaze position alone cannot say which car was looked at. During pursuit, however, the eye's velocity follows the target's velocity, and a constant calibration offset drops out when you take the difference over time. Matching the velocity profile of a pursuit segment to the velocity of short candidate tracks lets us attach a human judgment ("that thing is a car and it moves") to one specific track. Static clutter from parked cars and false motion from tall buildings appearing to shift as the aircraft flies past are exactly the two failure modes current pipelines cannot resolve without region-specific engineering.

Core Claim

Oculomotor features, specifically the separation of fixations, saccades, and smooth pursuit plus their velocity profiles, can be converted into target-region point labels that are accurate enough to close a large part of the geographic generalization gap of wide-area motion detectors. Because analysts generate these labels as a byproduct of work they already do, the extra human time is zero. We claim this beats unsupervised self-training on the target region. We also claim it reaches label efficiency comparable to explicit clicking at a fraction of the active effort. Gaze adds information that motion cues alone lack in two cases. Analysts keep looking at a car after it stops at an intersection, which preserves its identity. Analysts do not pursue rooftops that only appear to move, which rejects parallax.

Where This Sits in Prior Work

The motivating evidence comes from two wide-area motion detectors, both of which work but were never moved to a new region. Multi-Cue Vehicle Detection for Semantic Video Compression in Georegistered Aerial Videos fuses a motion signal computed over two image dimensions plus time (a flux tensor) with a YOLOv3 appearance detector and a building-parallax mask on the Albuquerque (ABQ) dataset. It reports 69.7% precision and 70.1% F-measure, against 26.9% precision for motion alone and 9.4% for appearance alone. Its georegistration and masking are tuned to one flight. Transforming Temporal Embeddings to Keypoint Heatmaps for Detection of Tiny Vehicles in Wide Area Motion Imagery (WAMI) Sequences proposes candidate regions with a light CNN and learns discrete motion codes over 5-frame windows with a vector-quantized autoencoder. A Transformer then maps these codes to Gaussian keypoint heatmaps, reaching precision up to 0.98 and the best F1 in four of six areas of WPAFB 2009, all within one city. We reuse this keypoint formulation as our detector. It suits gaze-derived labels because a point label maps directly onto a Gaussian heatmap target. It also avoids the box-overlap collapse that Detecting tiny objects in aerial images: A normalized Wasserstein distance and a new benchmark documents, where a 3-pixel shift drops the overlap score of a 5×8 vehicle from 0.54 to 0.14.

The geographic-shift literature constrains the design. Position: Mission Critical – Satellite Data is a Distinct Modality in Machine Learning reports drops of up to 30% at unseen sites and argues for geographically blocked evaluation, which we adopt. Adapting Vehicle Detectors for Aerial Imagery to Unseen Domains with Weak Supervision adapts static aerial vehicle detectors across regions by fine-tuning a latent diffusion model and extracting pseudo-labels from its cross-attention. It reaches 75.4% AP50 from New Zealand (LINZ) to Utah (UGRC). It works on 112×112 static tiles, which the authors note blurs very small objects, and it uses no motion. Statistically-Guided Deep Network Transformation to Harness Heterogeneity in Space (Extended Abstract) splits networks into shared and region-specific branches based on spatial error clusters. It needs grid-aggregated data and dense labels, which do not fit sparse point targets. Our proposal instead supplies the missing ingredient those methods assume: some target-region supervision.

The oculomotor literature supplies the method and one close neighbour. SaFiDe and velocity-threshold segmentation give us fixation and saccade parsing. Pursuit separation follows Kasneci et al.. The closest system is GLANCE: Gaze-Led Attention Network for Compressed Edge-inference, which uses wearable gaze to crop regions for a detector at inference time. On COCO it raises small-object accuracy from 39.2% to 51.3%. GLANCE uses gaze as a run-time attention window in egocentric indoor scenes. We use gaze as a training signal for cross-region adaptation of overhead video detectors, and we exploit pursuit dynamics, not just where the eye rests. Expertise Classification of Soccer Goalkeepers and A naive Bayesian method for eye movement recognition of map linear elements show that gaze sequences encode what a viewer is doing on scenes and maps. Both classify the viewer rather than locating objects in the world. That reversal is our contribution.

Research Program

We first collect gaze data, since no gaze-on-WAMI dataset exists in the file. Eight to twelve participants, including some with imagery-analysis experience, view stabilized WPAFB 2009 and ABQ sequences on a screen-based eye tracker. They perform a natural monitoring task: report traffic events by voice, with no instruction to look at cars. Each session is capped at a fixed duration per region.

Second, we parse the gaze and build labels. Raw gaze is split into fixations, saccades, and pursuit segments. In parallel, the source-trained detector produces low-threshold candidate keypoints in each target window, which are linked into short tracks. A pursuit segment $g$ with gaze velocity $v_g(t)$ is assigned to the candidate track $c$ that maximizes a match score.

$$s(g,c)=\exp\!\left(-\frac{1}{T}\sum_{t}\frac{\lVert v_g(t)-v_c(t)\rVert^2}{2\sigma_v^2}\right)\cdot \mathbb{1}\left[\lVert \bar p_g-\bar p_c\rVert<r\right]$$

Here $v_c(t)$ is the track's image-plane velocity, $T$ is the segment length in frames, $\sigma_v$ is a velocity tolerance set from pursuit gain statistics, $\bar p$ denotes mean positions, and $r$ is a coarse spatial gate sized to the tracker's error. Pursuit episodes that are matched become positive moving-vehicle labels. Long fixations on a track that recently matched a pursuit become positive stopped-vehicle labels. Candidate tracks with apparent motion that pass through the analyst's central vision without drawing pursuit become soft negatives, weighted by how long they were near the gaze. These soft negatives are the mechanism for rejecting parallax.

Third, we adapt the detector. The keypoint detector is fine-tuned on the target region using these labels, with label confidence taken from $s(g,c)$. A consistency term on unlabelled frames prevents collapse.

Fourth, we analyze the labels. We measure precision and recall of gaze labels against ground truth as functions of viewing time, expertise, and vehicle speed. This tells us when pursuit is reliable and when it is not. Slow vehicles may not elicit pursuit, which is a known limit of the signal.

Baselines

The naive reference point is the source-only detector: the keypoint model of Transforming Temporal Embeddings to Keypoint Heatmaps, trained on WPAFB and applied to ABQ unchanged, and the reverse. It defines the size of the geographic gap. The file reports no cross-region number for any WAMI detector, so this gap is a first measurement in its own right.

The strongest existing in-region system is the fused flux-tensor, YOLO, and parallax-mask pipeline of Multi-Cue Vehicle Detection. It reports 69.7% precision and 70.1% F-measure on ABQ when built for ABQ. We re-implement it, configure it on WPAFB, and transfer it to ABQ with only re-estimation of the georegistration. We also report it in-region as an upper reference. Beating its transferred version is required. Approaching its in-region 70.1% F-measure with gaze-only target supervision would be a strong result.

The strongest annotation-free adaptation competitor is motion-consistency self-training. Here the source detector's confident, temporally consistent tracks on the target region serve as pseudo-labels, using the same detector, the same number of fine-tuning steps, and the same hyperparameter search budget as our method. This baseline directly tests whether humans add anything.

The strongest weak-supervision competitor is diffusion-based pseudo-labelling as in Adapting Vehicle Detectors for Aerial Imagery to Unseen Domains with Weak Supervision, which reports 75.4% AP50 for LINZ to UGRC with Faster R-CNN. It is static, so we run it on the target frames, train on its labels, and then add the same temporal head.

Ablations isolate the key ingredient. In the first, gaze position replaces velocity matching (nearest candidate to the fixation centroid). In the second, pursuit is removed and only fixations are used. In the third, gaze traces are shuffled across time. In the fourth, gaze is simulated as ground-truth positions with matched noise and no human judgment, which tests whether the human is doing the deciding or merely pointing. A win means higher F1 than self-training and the transferred fused pipeline at the same detector, compute, and tuning budget, with the gain concentrated on stopped vehicles and fewer parallax false positives.

Why not just…?

Why not just have the analyst click on vehicles? Clicks are cleaner than gaze, and this is the most serious threat. The argument for gaze is that it costs no extra time, while clicks interrupt the task. We will settle this with an effort-matched comparison. A separate group clicks vehicle points for budgets of 1, 5, and 15 minutes, and we plot target F1 against active human time. If a few minutes of clicking beats a full session of passive gaze, our claim narrows to "gaze is a free supplement," and we would test whether gaze plus a small click budget beats clicks alone.

Why not just self-train on the target region using motion consistency? Self-training inherits the source model's systematic mistakes. In ABQ these are rooftop parallax, which looks like coherent motion, and stopped vehicles, which lose their motion signal. Multi-Cue Vehicle Detection shows motion-only precision of 26.9% from parallax. Confident pseudo-labels would reinforce those errors rather than fix them. Our breakdown of false positives per square kilometre over tall-building areas, and recall on stopped vehicles, tests this directly.

Why not just use gaze as a run-time attention window, as GLANCE does? GLANCE shows that cropping around gaze helps small objects. However, it only helps where the analyst happens to be looking, and it needs a human present at every inference. Our approach uses gaze once per region to produce a detector that then runs unattended over the whole frame. We include a GLANCE-style variant, with gaze crops fed to the source detector, as an extra comparison.

Why not just use an open-set foundation detector that needs no adaptation? Adapting Vehicle Detectors for Aerial Imagery to Unseen Domains with Weak Supervision reports that its adapted model beats open-set detectors by more than 50% on aerial vehicles. Such models also have no motion reasoning for 5-pixel targets, so we do not expect them to compete. We report one as a sanity point.

Proposed Experiments and Decisive Checks

The main experiment adapts WPAFB 2009 to ABQ and ABQ to WPAFB under geographic blocking. Detections are scored as keypoints matched within a fixed pixel radius. We report precision, recall, and F1 separately for moving vehicles, stopped vehicles, and parallax-prone tiles. We also report the fraction of the source-to-in-region gap recovered.

A secondary static check uses LINZ to UGRC frames, where we collect gaze on still tiles. Without pursuit, this tests whether fixation-only gaze still helps where motion is absent. We compare against the 75.4% AP50 reported for diffusion-based adaptation.

The decisive checks are four. First, the gap must exist: source-only F1 must fall clearly below in-region F1. If it does not, the problem is not real for WAMI. Second, velocity matching must beat position-only matching in label precision. This is the specific mechanism we claim. Third, gaze-adapted F1 must exceed self-training at matched compute, with the advantage concentrated on stopped vehicles and parallax regions. Fourth, the real-gaze model must beat simulated gaze at matched noise. Otherwise the human is contributing only a location prior and not a judgment. Failure of the second or fourth check would mean eye movements are not adding what we claim, and we would report that as a negative result about pursuit-based supervision.