Self-training adapts a trajectory forecaster to a new city without labels. A "teacher" model predicts future paths for unlabeled scenes, and those guesses become training targets, or pseudo-labels, for a "student." These guessed futures do not go wrong one step at a time. Typically the teacher picks a slightly wrong speed or heading early on, and the error then compounds, so the whole tail of the path drifts off together. Current methods treat a guessed trajectory as one unit: they keep it or drop it, then score every future step equally. Two fixes from the forecasting literature pull in opposite directions. One discounts distant steps because they are noisy. The other up-weights distant steps because the endpoint matters most. Under self-training the second fix amplifies exactly the most corrupted targets.
We propose a third option. A small network learns to predict, for each scene, the full error covariance of the teacher's guessed trajectory: how large the error is at each step and how strongly errors at different steps move together. The student is then trained with a Mahalanobis loss, meaning the squared error measured after "whitening" by that covariance. When the teacher's errors behave like drift, whitening turns the loss into a penalty on step-to-step increments. The student learns the teacher's turning and acceleration profile along the whole horizon but is not pulled toward the teacher's probably-wrong absolute endpoint. In two sentences: pseudo-label noise in forecasting is correlated across time, so the right weighting is a per-scene inverse covariance rather than a per-step scalar; with drift-like errors this keeps supervision on the shape of the path and drops it on the location of the endpoint.
Core Claim
Several lines of work motivate the gap, each stopping short of it. STraj adapts trajectory forecasters from Miami to Pittsburgh on Argoverse. It filters whole pseudo-trajectories by consistency across training epochs and by directional (cosine) similarity, but it then applies a uniform loss over all future steps. PTCL shows, for dynamic-graph node classification, that pseudo-labels get noisier the farther they sit from the labeled anchor. It handles this with a hand-tuned temporal curriculum that is sensitive to its decay hyperparameters. Learning under Temporal Label Noise models time-varying label flips with a matrix $Q(t)$ shared by all series, and only for discrete classes.
On the weighting side, the method we borrow comes from Quadratic Direct Forecast (QDF). QDF replaces mean squared error with a learned inverse-covariance quadratic loss, but its covariance is one static matrix for the whole dataset, and it targets clean labels. We use QDF's loss form, condition the covariance on each input, and apply it to the teacher's errors rather than to label autocorrelation.
The two weighting traditions act as constraints on the design. Discounting distant steps, as in IntentNet and the hierarchical motion autoencoder, is reported to starve long horizons of supervision. DistilTS calls this the seesaw effect. Escalating weight on distant steps, as in RSBG, is designed for clean labels.
Our claim is that both traditions are special cases of a diagonal weighting, and that the off-diagonal structure of pseudo-label error is what removes the conflict between them. The student loss on an unlabeled scene $x$ is
$$ \mathcal{L}_{\text{pseudo}}(x) = (\hat{y}_s - \tilde{y})^\top \, \Sigma_\phi(x)^{-1} \, (\hat{y}_s - \tilde{y}), $$
where $\hat{y}_s$ is the student's predicted future (positions stacked over $T$ steps), $\tilde{y}$ is the teacher's pseudo-trajectory, and $\Sigma_\phi(x)$ is the teacher-error covariance predicted by a small network with parameters $\phi$.
The intuition comes from a pure drift model. Suppose the teacher's error follows a random walk, $e_t = e_{t-1} + \eta_t$, where $\eta_t$ is independent noise with variance $\sigma^2$. Then the inverse covariance is a first-difference operator, and the loss reduces to
$$ \sigma^{-2}\sum_t \big((\hat{y}_{s,t}-\hat{y}_{s,t-1})-(\tilde{y}_t-\tilde{y}_{t-1})\big)^2 . $$
This is the textbook precision matrix of a random walk. The student is supervised on how the path moves at every step, including late steps, while the accumulated offset that the teacher most likely got wrong carries no weight. Real errors will sit between independent and pure drift, and $\Sigma_\phi$ is meant to interpolate between these cases scene by scene.
Research Program
We first estimate $\Sigma_\phi$ on the labeled source city, where we can compute the teacher's actual errors. A light head on the teacher's scene encoding outputs a lower-triangular Cholesky factor, a parameterization taken from QDF. It is trained by Gaussian negative log-likelihood of the teacher's top-mode error. Argoverse forecasters output several candidate futures. We pseudo-label with the teacher's highest-scoring candidate and fit the covariance to that candidate's error, so multimodal mismatch is absorbed as large, correlated variance rather than ignored.
The obvious objection is domain shift: a covariance fitted in Miami may be miscalibrated in Pittsburgh. We address this with a single per-step scale correction fitted on the target city using label-free signals. The first signal is the disagreement between teacher predictions across epochs and across dropout samples, which is the same signal STraj's update strategy already computes. The second is a short-horizon hindsight check: the teacher forecasts from the first half of each observed history, and that forecast is compared against the second half, which is known. Target-city ground truth is used only to analyze calibration, never for training.
Training otherwise follows STraj, with LaneGCN as the backbone, MIA→PIT, and STraj's pseudo-label update strategy kept as the trajectory-level filter. Only the per-step loss is swapped, so any gain is attributable to the weighting.
The analysis has two parts, and the first determines whether the idea is worth pursuing. We measure, on held-out target ground truth, the step-to-step autocorrelation of the teacher's pseudo-label errors. We also measure how much of the error variance a drift model explains. If target errors are close to independent across steps, the off-diagonal terms are irrelevant, and the proposal collapses to per-step variance weighting. We will report that as a negative finding. The second part checks calibration: the negative log-likelihood of true target teacher errors under $\Sigma_\phi$ before and after the scale correction, compared against a static dataset-wide covariance.
Baselines
The naive reference points are the source-only LaneGCN evaluated in Pittsburgh and plain self-training with uniform per-step loss and no filtering. They show how much any pseudo-labeling buys.
The strongest existing method is STraj, which reports reducing Miss Rate on MIA→PIT with LaneGCN from 0.755 to 0.685 and improving minADE and minFDE (the best-of-k average and final displacement errors) by 15–18%. We will rerun STraj ourselves with identical teacher checkpoints, unlabeled data, number of self-training rounds, and hyperparameter search budget. Beating it means lower Miss Rate and minFDE averaged over at least three seeds, with the gap exceeding seed variance. We expect minADE to move least and minFDE most.
Four ablations isolate the key ingredient, and each is dropped into STraj's pipeline. The first is fixed geometric discounting $\gamma^t$, with $\gamma$ tuned on the same budget. The second is exponential terminal escalation as in RSBG. The third is per-step diagonal weighting using the predicted variances only, which keeps $\Sigma_\phi$'s diagonal and zeroes the rest; this isolates the correlation structure. The fourth is a single static full covariance fitted on source data, in the style of QDF, which isolates the per-scene conditioning.
An oracle that weights with the true target error covariance bounds what any covariance-based weighting could achieve. As a secondary regime, we will run the same comparison in a semi-supervised setting with 10% of Miami labels to check that the effect does not depend on domain shift.
Why not just…?
Why not just discount distant pseudo-label steps with a fixed $\gamma^t$? This is the most serious threat, because if noise variance grows smoothly with horizon, a tuned discount approximates the diagonal of $\Sigma^{-1}$. Our argument is that discounting removes late-step supervision entirely, including the late-step turning and speed information that drift does not corrupt. DistilTS reports this kind of long-horizon starvation in distillation. The tuned-$\gamma$ baseline settles the question directly. If it matches our method on minFDE, the correlation story is wrong for this domain, and we will say so.
Why not just weight each step by the teacher's predicted variance? This is the diagonal ablation. It handles horizon-growing noise and differences between scenes but treats a 3 m endpoint offset as independent evidence at every step. Under drift, it therefore still penalizes the student many times for one shared error. If the autocorrelation analysis shows weak cross-step correlation, this baseline should tie with ours, and that outcome is a planned kill condition.
Why not just filter harder at the trajectory level and keep a uniform loss? STraj already does this well, and it is our base pipeline. Whole-trajectory accept/reject cannot keep the reliable early and shape information of a path whose endpoint drifted. APL shows in temporal localization that a single score per pseudo-label discards correct-but-low-confidence examples. A sweep of STraj's filtering threshold, reported alongside our method, will show whether stricter filtering alone closes the gap.
Why not just truncate pseudo-trajectories after the first second? Truncation is the extreme of discounting and loses all terminal supervision. It costs nothing to run, so we include it within the $\gamma$ sweep with $\gamma \to 0$ beyond the cut point.
Decisive Checks
The idea stands or falls on three measurements. First, target-city teacher errors must show substantial cross-step correlation; otherwise we stop. Second, on held-out target errors, the predicted per-scene covariance must have lower negative log-likelihood than the static covariance. Third, full-covariance weighting must beat both the tuned geometric discount and the diagonal-variance ablation on Pittsburgh minFDE and Miss Rate, under matched tuning budgets, by more than seed variance. A further prediction is that the gain should concentrate in scenes where the teacher's cross-epoch disagreement is large but heading agreement is high. Those are drifted-but-shaped pseudo-labels, and confirming this pattern would show the mechanism works as claimed rather than as generic regularization.
References
- STraj: Self-training for Bridging the Cross-Geography Gap in Trajectory Prediction
- PTCL: Pseudo-Label Temporal Curriculum Learning for Label-Limited Dynamic Graph
- Learning under Temporal Label Noise
- Quadratic Direct Forecast for Training Multi-Step Time-Series Forecast Models
- IntentNet: Learning to Predict Intention from Raw Sensor Data
- Efficient convolutional hierarchical autoencoder for human motion prediction
- Distilling Time Series Foundation Models for Efficient Forecasting
- Recursive Social Behavior Graph for Trajectory Prediction
- Towards Adaptive Pseudo-Label Learning for Semi-Supervised Temporal Action Localization