Fisher-Warped Time: Spending a Model's Temporal Resolution Where the Task Actually Changes

Models deployed over years lose accuracy because the world drifts. On the Wild-Time benchmark, standard models trained on past years lose about 20% of their performance on future years. A common fix is to give the model the timestamp, encoded as a smooth vector of sines and cosines, so it can learn a different input-to-output mapping for each moment. Calendar time, however, is a poor ruler for drift. A news classifier may see its label semantics shift sharply within one election year and then barely change for three years. If the model has limited capacity, meaning limited width and a limited number of time frequencies it can resolve, then encoding raw calendar time spreads that capacity evenly across the timeline instead of concentrating it where the mapping changes.

We propose to re-parameterize time by how much the task itself has changed. For each time slice, we compute a Fisher information task embedding: a fingerprint of which parameters of a fixed probe network the task is sensitive to, obtained from squared gradients of the log-likelihood. We then define a "functional clock" whose ticks are distances between consecutive fingerprints, and we feed this warped time, rather than the calendar date, into the model's temporal positional encoding. The central, falsifiable claim ties the three directions together. Fisher-warped time should improve future-year generalization mainly when the model is capacity-limited. The gain should shrink as width grows, because a large enough model can learn the warp from raw time by itself. The gain should also exceed what a warp built from input statistics alone provides whenever drift changes the labeling rule rather than only the inputs.

Core Claim

Let $e_s$ be the normalized diagonal-Fisher embedding of time slice $s$, computed on a frozen probe, and let $d$ be the symmetric cosine distance defined in Task2Vec. The functional clock at the end of slice $t$ is

$$\tau(t) = \sum_{s \le t} d(e_{s-1}, e_s),$$

the path length the task has traveled in Fisher space. Within a slice, $\tau$ is interpolated linearly in calendar time. The prediction model receives the standard sinusoidal encoding of $\tau(t)$ in place of the encoding of $t$.

For future deployment times $t > T$, where no labels exist, we estimate $\tau(t)$ in two ways. The first extrapolates the recent Fisher velocity, $\hat\tau(t) = \tau(T) + \hat v\,(t-T)$, where $\hat v$ is the average path length per unit time over the last few slices. The second computes a label-free Fisher embedding of the incoming unlabeled batch by sampling labels from the model's own predictive distribution, then places the batch at the clock position of its nearest point on the training trajectory plus its distance beyond the end. The claim is not that Fisher embeddings are better task descriptors in general; Task2Vec already shows that. The claim is specific: a fixed temporal-encoding capacity is spent better along a functionally warped axis, and this advantage is a capacity-bottleneck phenomenon that disappears at large scale.

Where This Sits in Prior Work

Four lines of work motivate the gap. Wild-Time establishes that real temporal shift is longitudinal and severe, and that classical continual-learning methods such as EWC and A-GEM often fail to beat plain ERM there. It does not condition models on time in any structured way or study capacity.

Fresh in memory shows that language models trained in stages spontaneously build a linear "recency axis" in their activations, a result that depends on adaptive optimizers. This is evidence that networks do spend representational capacity on a time coordinate. It does not tell us whether that coordinate is well allocated.

On Understanding of the Dynamics of Model Capacity in Continual Learning shows that effective capacity degrades in proportion to shift magnitude. Capacity-Constrained Continual Learning proves, for linear-Gaussian systems, that a finite memory budget should be "water-filled" according to how fast the underlying dynamics move. We read our warp as an empirical, nonlinear analogue of that allocation rule: spend resolution in proportion to rate of change.

Two papers supply the method. Task2Vec gives the probe-based diagonal-Fisher embedding and the distance we use. Continuous sinusoidal time encodings, as in Transformer Networks for Trajectory Forecasting and the periodic time-delta encoding in Temporal Inductive Path Neural Network, supply the encoding that we feed a warped rather than a raw coordinate.

The closest existing combinations each use Fisher information for a different job. Task-level Mixture-of-Experts routes whole tasks to expert modules using Fisher fingerprints. It has no notion of order or time, and nearest-neighbor routing cannot place a future slice beyond the last seen one; our continuous clock can. FIC-TSC uses the Fisher trace only as a flatness regularizer for time-series classifiers, never as a coordinate. Toward Understanding Catastrophic Forgetting uses Task2Vec to measure sequence complexity and finds that shrinking the network amplifies the damage done by complex sequences. That result supports our capacity prediction.

The same paper also reports a result that cuts against us: adjacent-pair Fisher distance ("sequential heterogeneity") has zero or negative correlation with error, for example $r=-0.25$ for Synaptic Intelligence on CIFAR-10. If adjacent Fisher distance is uninformative, a clock built by summing adjacent distances may be too. We take this seriously in two ways. First, that result concerns backward forgetting on shuffled image tasks, whereas we care about forward generalization on real timelines. Second, our first experiment is a gate that tests whether Fisher displacement predicts future-year degradation on Wild-Time before any model is built on it.

Research Program

The work proceeds in three stages.

The first stage is diagnostic. On each Wild-Time dataset (Yearbook, FMoW-Time, MIMIC-IV, HuffPost, arXiv), we compute per-year Fisher embeddings with a pretrained probe, using ResNet-34 for images and a DistilBERT-sized encoder for text and clinical records, each with a task-specific linear head. We also compute embeddings with a randomly initialized probe, following the observation in AutoTransfer that Fisher geometry on untrained models already carries task signal. We then ask whether the Fisher distance between the last training year and each test year predicts the ERM model's accuracy drop better than calendar gap or input-feature distance does. Input-feature distance here means cosine distance of mean frozen-encoder features. If Fisher distance does not beat calendar gap on this correlation, the program stops, or narrows to the datasets where it does.

The second stage builds the time-conditioned model. The Wild-Time backbone for each dataset receives a sinusoidal time encoding with $K$ frequencies, injected through feature-wise scaling and shifting in the penultimate layers. We compare three inputs to that encoding: raw calendar time, the input-statistics clock, and the Fisher clock. Everything else is held fixed. Both future-time estimators, velocity extrapolation and label-free Fisher placement, are evaluated.

The third stage tests the capacity claim directly. We sweep backbone width from 0.25× to 4× the Wild-Time default and sweep $K$ from 2 to 32. We then measure how the Fisher-over-calendar gain changes with capacity. The prediction is a gain that is largest at small width and small $K$ and approaches zero at the largest settings. A flat gain across widths would mean the warp is merely a better feature, not a capacity allocation, and that would falsify the central claim even if accuracy improves.

To separate concept drift from covariate drift, we add two controlled streams built from Yearbook images. In the first, the images are fixed and the labeling rule changes at non-uniform speeds; one example is a binary attribute defined by a threshold on a learned direction that rotates. In the second, the rule is fixed and the image era advances. Fisher warping should win clearly on the first stream and roughly tie the input-statistics clock on the second.

Baselines

The do-nothing reference is Wild-Time's ERM backbone trained on all past years with no time input, under the benchmark's Eval-Fix protocol. The file reports its degradation: Yearbook accuracy falls from about 98% in-distribution to about 79% out-of-distribution, MIMIC-IV AUC falls from 0.90 to 0.72, and the average drop across datasets is 20% (Wild-Time). No reported per-method result was found on Lacuna for HuffPost, arXiv, or FMoW-Time, so on those datasets we re-run ERM ourselves. We treat the strongest Wild-Time method on each dataset, re-run under our splits, as the strongest existing method. Beating it means a higher out-of-distribution average and higher worst-year performance, averaged over at least three seeds, with non-overlapping confidence intervals on at least three of the five datasets.

The most important baseline is the same backbone with a raw calendar-time sinusoidal encoding. It has identical parameters, frequency budget, tuning grid, and training epochs. It isolates whether warping, rather than time conditioning itself, drives any gain.

The key ablation replaces the Fisher clock with an input-statistics clock built from mean-feature distances on the same probe. It shares the same interpolation and extrapolation code and the same hyperparameter budget. It isolates the functional content of the Fisher embedding.

We also compare against adaptive-window training, which retrains ERM on a window of recent years chosen by the Goldenshluger-Lepski rule from Model Assessment and Selection under Temporal Distribution Shift. This is the principled version of the industry default "train on recent data," and the file reports no Wild-Time number for it.

A Fisher-nearest-neighbor expert baseline adapts Task-level Mixture-of-Experts to time. Each year's Fisher embedding routes to one of $M$ adapters, and $M$ is chosen so that the total parameter count matches our model. That paper reports a +2.6% average relative gain over multi-task BART-Base on CrossFit, but it reports nothing on temporal data.

Finally, a width-matched larger model receives raw time and is given the extra parameters that Fisher computation would otherwise "cost" in compute. All methods share data, epochs, and a fixed tuning budget of 20 configurations each, selected on the last training year held out as validation.

Why not just…?

Why not just encode raw calendar time and let the model learn the warp? This is the real threat, and our capacity prediction concedes it at scale. A sufficiently wide network with enough frequencies can learn any monotone warp from raw time. The width sweep is designed to show where that stops being true. If raw time matches the Fisher clock even at 0.25× width and $K=2$, the proposal fails, and we will report that.

Why not just use cheaper input-feature statistics to measure drift? Input averages capture what the data look like, not how inputs map to labels. Task2Vec shows that such averages cannot separate tasks that share an image domain but differ in labels, and that Fisher-based selection reaches +9.97% relative error over the optimum versus +30.18% for a generic expert. That evidence comes from static model selection, however, not from temporal data. On purely covariate drift, as in Yearbook's changing photo styles, we expect a tie. The controlled concept-drift stream and the input-clock ablation will show whether the Fisher cost pays off anywhere real.

Why not just retrain on a recent sliding or adaptive window? Windowing discards old data rather than organizing it. Adaptive Estimation and Learning under Temporal Distribution Shift shows that fixed windows face an unavoidable bias-variance trade-off under drift. Adaptive windows reduce that trade-off but still cannot exploit years that are distant in calendar time yet functionally close. Such recurrences are exactly the cases our clock places nearby. Adaptive windowing is run as a baseline, and gains concentrated on test years with functional recurrence would confirm the mechanism.

Why not just use EWC or other continual-learning methods? Wild-Time reports that these methods often underperform ERM on longitudinal shift. EWC also uses the Fisher information to freeze past behavior, which is the opposite of what forward generalization under drift needs. We include the best such method as part of the strongest-existing-method comparison rather than as a separate hypothesis.

Decisive Checks

Three results would settle the proposal.

The first is the stage-one gate. Fisher displacement must correlate with future-year degradation more strongly than calendar gap, on at least three of the five Wild-Time datasets. This speaks directly to the negative adjacent-distance result in Toward Understanding Catastrophic Forgetting.

The second is the capacity curve. The out-of-distribution gain of the Fisher clock over raw calendar time must decrease monotonically with width and with $K$, and be statistically non-zero at the smallest settings.

The third is the dissociation on the controlled streams. The Fisher clock must beat the input clock under label-rule drift and tie it under covariate drift.

We will also report robustness to the choice of probe, pretrained versus random, given the probe sensitivity documented for Task2Vec. We will report the wall-clock cost of computing Fisher embeddings, one backward pass per example on a frozen probe per slice, relative to training. And we will report whether the label-free placement estimator matches velocity extrapolation. If it does, the method can be used at deployment without future labels.