A point tracker follows a physical spot on a surface through a video. When that spot goes behind another object, the tracker no longer has anything to match, and the true answer becomes a spread of possible hidden paths. Current trackers handle this badly in one of two ways. Fast deterministic trackers output one coordinate per frame, and under occlusion that coordinate is the average of all plausible paths, which can cut straight through the occluder. Generative trackers instead sample many plausible paths. They are better at this, but they pay for it three times: many integration steps per sample, an outer refinement loop, and a best-of-N search over candidates. We propose to spend generative effort only where it is needed. A deterministic tracker handles visible frames as usual. Wherever it predicts an occluded span, a one-step generator draws a batch of hidden paths in a single parallel forward pass, conditioned on where the point entered and, when available, where it reappeared.
The approach has two parts, and the second matters as much as the first. Because the true probability of a hidden path given the video cannot be computed for these models, we evaluate them with a sample-based calibration score. That score tells us whether the sampled paths are too narrow (overconfident) or too broad (underconfident). It also tells us whether collapsing generation to one step quietly kills the diversity that justified using a generator at all. The claim we test is concrete. On occluded spans, a gated one-step sampler matches the calibration and amodal accuracy of a multi-step generative tracker, and does better than a cheaper multi-hypothesis regression head. Its latency stays close to that of a deterministic tracker.
Where this sits relative to prior work
The closest work is Generative Point Tracking with Flow Matching. It recasts tracking as flow matching, meaning a generator that learns a velocity field transporting noise into whole trajectories. It keeps the iterative refinement of discriminative trackers, conditions each sliding window on the previous one, and picks among N samples using predicted confidence. It motivates our proposal in two ways: it shows multimodal output helps under occlusion, and its authors note that best-of-N is costly. We differ in three respects. GenPT generates entire trajectories with multi-step integration. We generate only the occluded spans, in one step, with all candidate samples in one batch. GenPT reports discriminative metrics (Average Jaccard, position accuracy). We additionally measure whether its posterior over hidden paths is calibrated.
The one-step machinery comes from Mean Flows for One-step Generative Modeling. MeanFlow trains a network to predict the average velocity over a whole time interval rather than the instantaneous one, so a single step covers the full noise-to-data path without a teacher. It reports a 1-step FID of 3.43 on ImageNet 256×256. 3D MeanFlow extends this to point sets and adds a gate that runs completion only on the detection proposals that benefit. That gating idea is the direct template for running our sampler only on occluded spans. 3D MeanFlow also constrains our design: its authors report that one-step generation loses diversity relative to multi-step sampling. This is exactly the failure that would make a generative tracker pointless, so we measure it rather than assume it away.
The deterministic side supplies the backbone and the reference points. TAPIR combines global matching, which can re-find a point after occlusion, with parallel temporal-convolution refinement, and runs at about 150 FPS for 50 points. We reuse its structure for visible frames and its re-detection to supply exit points. Track-On uses memory banks to carry points through occlusions online, and TrackIME prunes the search to the tracked instance. Both still output a single path. Our proposal differs from these trackers by representing a distribution only on hidden frames. It differs from the earlier joint flow-and-occlusion estimator Iterative Residual Refinement by not unrolling refinement iterations at all; that paper notes its FLOPs stay proportional to iteration count even with shared weights.
The evaluation borrows MIRA. MIRA scores a conditional generator using only samples. It draws random regions around candidate samples and asks how often the true outcome falls inside them, which yields a reference value of 2/3 for a calibrated model: below means overconfident, above means underconfident. MIRA has only been tested on images, lensing, and MRI, never on trajectories. Two further papers motivate why scalar metrics are insufficient here. Assessing Generative Models via Precision and Recall separates sample realism from mode coverage. Flow-GAN shows that sharp-looking samples can coexist with very poor distributional fit.
Core Claim
Occlusion makes the tracking problem multimodal only on a minority of frames. Generative sampling should therefore be confined to those frames and compressed to a single parallel step. Whether this compression preserves the hidden-path distribution can be checked directly with a likelihood-free calibration score. We expect a one-step average-velocity sampler, conditioned on entry state, image features, and exit observation, to stay near MIRA's 2/3 reference on occluded spans. We expect a K-way regression head to be overconfident there, and uniform step-truncation of a multi-step tracker to lose recall.
Research Program
We first build the deterministic stage: a TAPIR-style tracker, trained on Kubric, that outputs positions and occlusion probabilities. Thresholded occlusion predictions define occluded spans. Each span carries an entry state, which is the last confident visible positions and velocities, and optionally an exit state, which is the re-detected position after reappearance. Exit states exist in the offline setting; in the online setting the exit is unknown and the posterior is wider.
The second stage is a span sampler. It takes the span length, the entry and exit states, and cost-volume features pooled along the span. It is trained with the MeanFlow objective to map Gaussian noise to the hidden trajectory in one step. The training targets come from synthetic data where hidden positions are known: Kubric, PointOdyssey, and Dynamic Replica. One further cheap term penalizes samples that land on pixels the frame shows as unoccluded background. A point that is supposedly hidden cannot sit at a visible location that is not its own. This "visibility violation" also serves as an evaluation metric. All N samples per span are drawn in one batched pass, and there is no confidence-guided search.
The third stage is evaluation. We adapt MIRA to trajectories by defining regions as balls under the mean per-frame Euclidean distance over a span. We report the score separately for short and long spans and for online and offline conditioning. We also report trajectory-level precision and recall in the PRD sense. MIRA degrades with few test samples, so we pool spans within duration bins and report bootstrap intervals.
Baselines
The naive reference point has two forms. One is the deterministic tracker's own prediction on occluded frames. The other is constant-velocity extrapolation from the entry state, plus linear interpolation to the exit state when the exit is known. Matching these on occluded spans would mean the generator adds nothing.
The strongest existing generative competitor is GenPT, run with its full multi-step integration, refinement, and best-of-N search. It is the only method in the file that models multimodal hidden paths, and its authors report a lead on the sliding-occluder TAP-Vid variant they introduce. We found no reported numerical result on Lacuna for GenPT on that variant or on TAP-Vid-DAVIS, so we will rerun it from its release on identical splits and hardware. The strongest deterministic reference is Track-On, which reports AJ 65.0 on TAP-Vid-DAVIS (Track-On). Alongside it is TrackIME on TAPIR at AJ 65.3 (TrackIME). Our method must not fall below the deterministic backbone's AJ on visible frames. It beats GenPT if it matches GenPT's occluded-frame accuracy and calibration within bootstrap error at a substantially lower wall-clock cost per point.
Fairness requires several controls. All learned methods share the Kubric training data, a matched parameter budget (GenPT uses 12M), the same GPU, and the same hyperparameter search budget. Latency is measured end to end per point per frame, including the gate.
Three ablations isolate the key ingredients. First, the gate is removed and the one-step sampler generates whole trajectories, which tests whether confining generation to occluded spans matters. Second, a same-architecture sampler is trained with ordinary instantaneous velocity and run at 1, 2, and 4 steps, which tests whether the average-velocity target matters. Third, the exit-state conditioning is dropped, which separates offline from online difficulty.
Why not just…?
Why not just run GenPT with one or two integration steps? This is the cheapest fix. It is the instantaneous-velocity arm of our ablation, applied to GenPT itself. MeanFlow's comparison against few-step baselines on images (Shortcut-XL FID 10.60 and iCT-XL FID 34.24, against 3.43) suggests that naive truncation degrades badly. Trajectories are low-dimensional, though, and their paths may be nearly straight. If truncated GenPT matches our MIRA and recall numbers, the average-velocity component is unnecessary and we will report that.
Why not just add a K-hypothesis regression head to a deterministic tracker? This alternative is the real threat. It gives multiple paths in one pass at almost no cost and needs no generative training. Winner-take-all heads are known to leave some hypotheses unused and to produce a fixed number of modes with no notion of spread. MIRA is exactly the instrument that would expose this as a score below 2/3. Nothing in the file demonstrates that failure on trajectories, however, so the head is a primary baseline rather than a straw man. If it reaches calibration near 2/3 with comparable recall, our conclusion becomes that occluded-span multimodality needs only a cheap mixture head, not a generator. That would still be a useful, publishable answer.
Why not just distill GenPT into a one-step student? Distillation keeps the teacher's behavior and adds a training stage. 3D MeanFlow reports that multi-stage distillation causes drift on unordered sets, but that evidence comes from shapes, not tracks. We include a distilled student under the same compute budget. The test that settles it is whether the student inherits GenPT's calibration or becomes overconfident.
Decisive Experiments
The main test uses the sliding-occluder TAP-Vid variant. Because the occluder is artificial, the hidden positions under it are known from the original annotations. We add Kubric, PointOdyssey, and Dynamic Replica, where hidden positions are rendered. On occluded frames we report amodal position accuracy at the TAP-Vid pixel thresholds, best-of-N displacement, trajectory MIRA, precision and recall, and the visibility-violation rate. On TAP-Vid-DAVIS, Kinetics, RGB-Stacking, and RoboTAP we report standard AJ, position accuracy, and occlusion accuracy. These benchmarks do not label positions behind occluders, so they only check that visible-frame tracking is preserved.
Three outcomes would decide the question. The first is a calibration-versus-step-count curve from 1 to full integration, which shows whether one-step sampling loses recall on long occlusions, the gap 3D MeanFlow warns about. The second is a head-to-head on MIRA between the one-step sampler and the K-hypothesis head at equal latency. The third is a latency-versus-occluded-accuracy plot placing our method, GenPT, truncated GenPT, and TAPIR on one frontier.
The proposal succeeds if our method sits on that frontier near TAPIR's speed and GenPT's occluded-span accuracy, with MIRA closer to 2/3 than the regression head. It fails informatively if the head or truncated GenPT gets there first.
References
- Generative Point Tracking with Flow Matching
- Mean Flows for One-step Generative Modeling
- 3D MeanFlow: One-Step Point Cloud Completion and Generation via Average-Velocity Transport
- TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement
- Track-On: Transformer-based Online Point Tracking with Memory
- TrackIME: Enhanced Video Point Tracking via Instance Motion Estimation
- Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation
- MIRA: A Score for Conditional Distribution Accuracy and Model Comparison
- Assessing Generative Models via Precision and Recall
- Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models