Online subspace trackers are algorithms that keep a small, continuously updated set of directions summarizing a high-dimensional data stream, such as a sensor array, a video feed, or a robot's motion capture. They are almost always designed, tuned, and ranked on synthetic streams: a known low-dimensional subspace rotating on a schedule, white Gaussian noise, and entries that go missing independently at random. Real streams break every one of those conveniences. Missing entries come in bursts, noise is heavy-tailed and depends on distance or lighting, and the number of active directions changes over time. Nobody has checked whether the synthetic protocol actually predicts which tracker, and which step size, works best on real data. This proposal treats that as an empirical question with a measurable answer.
The claim has two parts. The first is that standard synthetic generators give tracker rankings and hyperparameter choices that transfer poorly to real streams. We can measure this without knowing the true real subspace by scoring each tracker on its ability to predict held-out real entries before seeing them. The second is that a real-anchored generator does better. This is a synthetic stream whose missingness bursts, noise tails, rank changes, and drift speed are fitted to a short prefix of a real stream, with drift speed measured by a subspace tracker itself. It should restore that transfer while keeping what synthetic data offers: a known ground truth and parameters you can vary. The test is modeled on the "Turing test for models" of Is this Model for Real?. A generator passes if the real stream's hyperparameter-versus-error landscape is no farther from the synthetic landscape than two synthetic replicates are from each other.
Why this gap is real
Four tracking papers motivate the study, because each validates mainly on convenient generators and touches real data only lightly. GROUSE runs stochastic gradient steps along the shortest paths of the space of subspaces (the Grassmannian). It is tested on Gaussian streams with random missingness and one chlorine-sensor dataset, and it reports that its step size has a narrow sweet spot outside which it diverges. NORST-miss adds a residual-based change test with near-optimal detection delay, proven under piecewise-constant subspaces and random missingness, and demonstrated on a single real video ("Meeting Room") without numbers. OR-PCA separates sparse corruptions from a low-rank basis online, and its phase transitions are mapped almost entirely on synthetic uniform corruptions with a known, fixed rank. Geodesic subspace estimation derives a $T=2k$ recovery threshold on Gaussian data and shows real video and fMRI only qualitatively.
Two lines of work constrain the design. The robotics reality-gap literature (The Reality Gap in Robotics, Quantifying the Reality Gap) finds that better simulator accuracy often fails to predict better real performance. That is a correlation between simulated and real outcomes that breaks down, and it is exactly the quantity we will measure for trackers. Simulator tuning finds that fitting 57 simulator parameters did worse than fitting 31, and that three parameters did most of the work. We take this as a warning to keep the anchored generator to a handful of interpretable parameters. Chang et al. find that behavioral structure in synthetic video mattered more than photorealism. On DanceTrack, multi-group synthetic scenes lifted tracking accuracy from 68.8 to 73.5 on the Higher Order Tracking Accuracy (HOTA) metric. We translate that lesson to streams: missingness bursts and rank changes should matter more than noise variance matching.
The closest methodological precedent is Rosenberg-Kima and Pardos, whose method we reuse. They compare the fit landscapes of real and simulated educational data over a 390,625-point grid, offline, for one fixed model. We apply the same landscape comparison to streaming algorithms, across algorithms, with a generator that is calibrated online rather than fixed.
Research Program
We first build a ground-truth-free real metric. At each time step we hide a random 10% of the observed entries of the incoming vector. We then score the tracker's prediction of those entries from its current basis, before it updates on the vector. This is the prequential imputation error:
$$E = \frac{1}{T}\sum_{t=1}^{T} \frac{\|P_{H_t}(v_t - U_{t-1} w_t)\|_2^2}{\|P_{H_t} v_t\|_2^2}.$$
Here $v_t$ is the stream vector at time $t$, $H_t$ is the set of hidden entries, and $P_{H_t}$ keeps only those entries. $U_{t-1}$ is the tracker's basis before the update, and $w_t$ is its least-squares fit on the visible entries. Because every tracker is scored on the same hidden entries, $E$ ranks trackers on real data without needing the true subspace.
Next we build the real-anchored generator. From the first 15% of each real stream we fit a small set of parameters. The first is a two-state Markov burst model of missingness, which has a switching rate and a burst length. The second is a Student-t noise tail index. The third is a rank trajectory, estimated from the singular value gap in sliding windows. The fourth is a drift speed, taken as the principal-angle velocity between successive GROUSE bases on the prefix, which gives a direct Grassmannian reading of how fast the real subspace turns. The generator then produces streams with a known rotating subspace that match these statistics. Because the parameters are named, each can be swept to create stress regimes that the prefix never showed.
The comparison uses a fixed pool of trackers: GROUSE, NORST-miss, OR-PCA, streaming heteroscedastic probabilistic PCA, and a sliding-window version of the geodesic estimator. Each is run over an identical hyperparameter grid covering step size, forgetting factor or window length, rank, and change-test threshold. We evaluate every tracker and configuration on the default generator, the anchored generator, and the real stream.
Baselines
The naive reference point is to use each paper's published default hyperparameters and to rank trackers by the numbers reported on their own synthetic data. This is what a practitioner does today, and any method must beat it on real imputation error.
The main incumbent is the standard synthetic protocol from the GROUSE and NORST-miss papers: Gaussian noise, independent random missingness, and a smooth or piecewise-constant rotation, with SNR and rank matched to the real stream's prefix. It is the strongest competitor because it is what the field actually uses and it already receives the obvious calibration. To keep the comparison fair, it gets the same grid, the same number of synthetic stream-steps, and the same prefix statistics that are fittable within its own model family. The anchored generator beats it if two things hold on at least four of six real streams. The rank correlation between synthetic and real configuration rankings must be higher. The real error of the synthetically selected configuration must also be lower.
No reported result found on Lacuna for prequential or held-out imputation error on the chlorine, Meeting Room, Curtain, Waterfall, OSSI, or DTTD streams. The only quantitative tracker comparisons in the file are synthetic. OR-PCA tolerated up to 50% sparse corruption where GRASTA failed near 25%, and it processed $10^8$ samples in about 15,910 seconds versus over 252,630 for GRASTA (OR-PCA). A central question of this study is whether that ordering survives on real streams.
The strongest non-synthetic competitor is prefix tuning: grid-search every configuration directly on the same 15% real prefix using $E$, then deploy. This uses exactly the data the anchored generator is fitted on.
A block bootstrap of the real prefix is a nonparametric alternative. It resamples contiguous chunks and preserves local structure without any model.
The ablations remove one anchored component at a time: bursts, tails, rank trajectory, and drift speed. This isolates which one carries the transfer. Our prediction, following Chang et al. and Collins et al., is that bursts and rank changes dominate.
Why not just…?
Why not just tune on the real prefix and skip synthetic data? This is a real threat. For choosing hyperparameters on streams that stay roughly stationary, prefix tuning may match or beat any generator, and if it does we will report that plainly. The generator can win in two places. The first is streams whose later regime differs from the prefix, such as the dim and dark lighting segments of DTTD or the illumination changes in Meeting Room. There, sweeping the fitted drift and burst parameters beyond prefix values can pick configurations that hold up later. The second is algorithm design and theory, which need ground truth that no real prefix provides. We measure both by splitting results by whether a detected change occurs after the prefix.
Why not just block-bootstrap the real data? A bootstrap preserves local texture but has no known subspace. It cannot extrapolate to faster drift or longer outages, and it cannot test a theoretical claim such as the $T=2k$ threshold of the geodesic estimator. If the bootstrap matches the anchored generator on ranking transfer, the conclusion is that fidelity matters and parametric structure does not. We would then narrow our claim to stress testing and ground-truth analysis.
Why not just add a generic robustness suite of heavy-tailed noise and bursty masks at arbitrary settings? An uncalibrated suite changes the rankings but gives no reason to trust the new rankings for a particular deployment. The comparison against the ablation with calibration turned off, which uses the same components at literature-default settings, answers whether fitting to the prefix matters or whether merely including the structure is enough.
Proposed Experiments
The real streams come from the file. They are the chlorine network from GROUSE, Meeting Room from NORST-miss, and Curtain, Waterfall, and OSSI fMRI from geodesic estimation. They also include depth streams from DTTD, whose sensor noise changes with distance and lighting, and high-rate video from chromatic-spikes. Where the motion-capture logs of simulator tuning are obtainable, we add paired commanded and measured Kinova trajectories. On those we also run GREAT, whose only published evaluation is an interpolated simulated airplane model.
We report three primary measures on the real stream after the prefix. The first is Kendall rank correlation between synthetic and real configuration rankings. The second is regret, meaning the real error of the synthetically chosen configuration minus the best real error on the grid. The third is the landscape test: the distance between the real and synthetic hyperparameter-error surfaces, compared against the spread between two independent synthetic replicates.
The decisive check has two outcomes. The anchored generator must pass the landscape test on streams where the default generator fails, and GROUSE's step-size optimum must move toward the real optimum under anchoring. If both happen, synthetic evaluation of trackers is repairable with a few fitted statistics. If neither does, the field's synthetic phase-transition results should be read as statements about the generator rather than about deployment.
References
- Is this Model for Real? Simulating Data to Reveal the Proximity of a Model to Reality
- Online identification and tracking of subspaces from highly incomplete information
- Provable Subspace Tracking From Missing Data and Matrix Completion
- Online Robust PCA via Stochastic Optimization
- Dynamic Subspace Estimation with Grassmannian Geodesics
- The Reality Gap in Robotics: Challenges, Solutions, and Best Practices
- Quantifying the Reality Gap in Robotic Manipulation Tasks
- Traversing the Reality Gap via Simulator Tuning
- On the Equivalency, Substitutability, and Flexibility of Synthetic Data
- Streaming Heteroscedastic Probabilistic PCA with Missing Data
- Digital Twin Tracking Dataset (DTTD): A New RGB+Depth 3D Dataset for Longer-Range Object Tracking Applications
- GREAT: Grassmannian Recursive Algorithm for Tracking & Online System Identification