Many practical problems ask us to learn the dynamics of dozens of similar systems at once: the brain networks of different fMRI subjects, a fleet of nearly identical machines, or a set of short video clips of the same kind of flickering texture. Each system is modeled as linear and time-invariant. Its hidden state evolves as $x_{t+1} = A x_t + \text{noise}$, and the goal is to estimate the matrix $A$ for every system. When a system's recorded trajectory is short or poorly varied, estimating its $A$ alone is unreliable. Multi-task learning, which trains related problems together so they share statistical strength, is the natural fix. The fix carries a known risk, though: sharing can also hurt. A system that does not resemble the others gets pulled toward the wrong answer. In deep multi-task learning this failure is called negative transfer.
We propose a joint estimator in which every system decides, direction by direction in its own state space, how much to borrow from the others. Linear systems make this decision computable. Each system's empirical state covariance, which we will call its Gramian, records exactly which state directions its data has explored and how well. We use it to set the strength of sharing automatically. A long, well-excited trajectory keeps its own estimate. A short one leans on structure shared across the group, and only in the directions where it has little data of its own. The target is an estimator that matches the sample-efficiency gains of current joint identification when the systems really are similar, and that is never meaningfully worse than estimating each system alone when they are not.
What exists and where the gap is
Three recent lines of work set up this problem. Multi-Task System Identification of Similar Linear Time-Invariant Dynamical Systems adds convex penalties to least-squares identification. These penalties encode shared sparsity, small deviations from a nominal model, or low-rank structure, and the paper shows that fewer fMRI scans are needed on Human Connectome Project data. Joint learning of linear time-invariant dynamical systems assumes each $A_m$ is a combination of $k$ unknown shared basis matrices. It proves a per-system error of $O(k/T + d^2k/(MT))$, where $T$ is trajectory length, $d$ is state dimension and $M$ is the number of systems, compared with roughly $O(d^2/T)$ for individual estimation. It also analyzes how this rate degrades under misspecification. Multi-task imitation learning for linear dynamical systems learns a shared low-dimensional representation across source controllers and then fits target controllers from few demonstrations.
These papers motivate the proposal, and they also leave the same gap. Each one fixes in advance how strongly systems are tied together. The first does it through hand-tuned penalty weights $\lambda$, chosen separately for each structural assumption. The second assumes exact membership in the shared span with $k$ known. The third freezes a shared representation. None of them lets the amount of sharing vary across systems or across state directions. Yet heterogeneity in data quality is exactly what the motivating applications show: in fMRI some subjects have many scans and some have few. The open problem these papers leave is negative transfer in system identification. It means guaranteeing that a well-measured system is not harmed by a pool of poorly measured or misfit neighbors, and that a poorly measured system borrows only what it needs.
The deep multi-task literature attacks negative transfer by reshaping gradients. Conflict-Averse Gradient Descent for Multi-task learning maximizes the worst-case task improvement within a ball around the average gradient. RotoGrad rebalances gradient magnitudes and rotates the shared features for each task. These methods are generic, which is why they work on deep networks where nothing better is known. Their mechanism is the one we argue against importing. In linear identification each system's information about $A_m$ is available in closed form, so negative transfer can be prevented statistically rather than by reshaping gradients during optimization.
Core Claim
The claim is that, for a family of related linear systems with uneven data, an empirical-Bayes estimator achieves three things together. It recovers the joint-learning rate when the shared structure holds, it stays within a small additive term of individual least squares when the structure fails for some systems, and it improves the worst-off system rather than only the average. The estimator shrinks each system toward a shared low-dimensional subspace, with the strength of shrinkage set by that system's own Gramian and by a single deviation scale estimated from the data. No current method gives all three properties at once, and none sets its sharing strength without manual tuning.
Research Program
The model writes the vectorized transition matrix of system $m$ as a shared part plus a private deviation:
$$\mathrm{vec}(A_m) = U\beta_m + \delta_m, \qquad \delta_m \sim \mathcal{N}(0, \tau^2 I),$$
where $U$ is a $d^2 \times k$ basis shared by all systems, $\beta_m$ are system-specific coefficients, and $\delta_m$ is the part of $A_m$ outside the shared span, with typical size $\tau$. Write $\Sigma_m = \sum_t x_t^{(m)} x_t^{(m)\top}$ for system $m$'s Gramian and $\sigma^2$ for the process-noise level.
Given $U$ and $\tau$, the posterior mean of $A_m$ has a simple closed form. It interpolates between the system's own least-squares estimate and its projection onto the shared span. The interpolation weight is matrix-valued, of the form $(\Sigma_m \otimes I + (\sigma^2/\tau^2) I)^{-1}$. In directions the system has explored heavily, the weight trusts its own data. In directions where $\Sigma_m$ is small, it defers to the shared span. Because the weight is a matrix, a system with a long trajectory that never excited one particular mode still borrows in that mode and only that mode.
Fitting proceeds by alternation. The algorithm estimates $U$ from all systems, weighting each one's contribution by its Gramian so that noisy short trajectories do not distort the basis. It then estimates $\tau^2$ and $\sigma^2$ by maximizing the marginal likelihood pooled across systems, and finally computes the per-system posteriors. The deviation scale $\tau$ is learned rather than tuned, and it carries meaning. If the systems really do share a span, $\tau$ goes to zero and the estimator reduces to the factorized method of Joint learning of linear time-invariant dynamical systems. If they share nothing, $\tau$ grows and each estimate approaches individual least squares. The rank $k$ is chosen by held-out one-step prediction on later time blocks of each trajectory.
The analysis has two goals. The first is a risk bound of the form "joint rate when $\tau$ is small, individual rate plus a lower-order term otherwise." It would extend the misspecification analysis of the Autom paper from a fixed estimator to an adaptive one. The second, more modest, is a characterization of when shrinkage helps the worst system. We do not promise either result. The empirical program is designed to stand on its own if the theory proves only partial. We restrict attention to stable systems, as prior theory does. Near-unit-root systems are tested empirically but not covered by the analysis.
Baselines
The naive reference points are individual least squares for each system and pooled least squares, which fits one $A$ to all data. These two bracket the problem. Beating individual least squares on average is easy for any sharing method. The meaningful test is matching or beating it on the worst-performing system, and beating pooling on heterogeneous families.
The strongest existing methods are the convex estimators of Multi-Task System Identification of Similar Linear Time-Invariant Dynamical Systems, specifically the nuclear-norm and small-heterogeneity variants, and the shared-basis factorization of Joint learning of linear time-invariant dynamical systems. For fairness, every method receives the same trajectories and the same held-out time blocks. Every method also gets an equal hyperparameter search budget: a grid over $\lambda$ for the convex methods and over $k$ for the factorization and for ours, all selected by the same held-out prediction criterion. The convex baselines are thereby given the tuning they need in practice, rather than an untuned default. On HCP fMRI the file gives only qualitative claims: fewer scans than least squares for comparable error, and better than LASSO for data-limited subjects. No reported result was found on Lacuna for numerical error or scan counts on HCP for any of these methods, so all numbers there will come from our own reruns. Beating these baselines means lower worst-system and median error at equal total data, with the gap widening as data becomes more uneven across systems.
A gradient-balancing baseline answers the obvious import from deep multi-task learning. We minimize the summed least-squares losses of the shared-basis model with CAGrad's update rule, using the method of Conflict-Averse Gradient Descent for Multi-task learning. On NYUv2 that method reports a 0.20% multi-task performance drop against 3.97% for PCGrad. We give it the same $k$ grid and tune its constraint radius $c$ on the same budget.
Two ablations isolate the key ingredient. The first replaces the Gramian-based matrix weight with a single scalar shrinkage per system. The second replaces it with one global scalar for all systems. If the scalar per-system version matches the full method, the direction-wise claim fails, and the contribution reduces to learning $\tau$ instead of tuning $\lambda$.
Why not just…?
Why not just tune the nuclear-norm penalty by cross-validation? This is the real threat. With a well-chosen $\lambda$, the convex low-rank estimator may match us whenever systems are equally well measured and equally close to the shared span. Our argument is that one global $\lambda$ cannot be right for every system when data quality varies. A setting strong enough to help a two-scan subject over-smooths a forty-scan subject. The synthetic sweep described below varies trajectory-length spread and the fraction of off-span systems independently, and it reports the worst-system error. If cross-validated nuclear norm matches us across that sweep, our method's only advantage would be convenience, and we will report it that way.
Why not just balance gradients with CAGrad or RotoGrad on the joint loss? These methods change the direction in which optimization moves. They do not change which solution the objective prefers. In a joint least-squares objective, a dominant well-measured system and an outlier system remain dominant and outlying at the optimum. CAGrad's documented strength is avoiding poor stationary points of non-convex deep losses. RotoGrad's feature rotations are designed for a large shared deep representation and do not apply naturally to identifying a matrix. The gradient-balancing baseline tests this directly. We expect it to move worst-system error less than per-system shrinkage does, because it lacks the information the Gramian supplies.
Why not just use a shared recurrent network or temporal transformer? The file's evidence suggests this fails in our regime of short trajectories. In Multi-task imitation learning for linear dynamical systems, learning without a shared low-dimensional linear representation stayed unstable even with 20 demonstrations, whereas the shared representation was stable with 2 to 5. We include a shared-trunk gated recurrent network on the fMRI experiment only, as a sanity check on held-out prediction, without expecting it to win.
Proposed Experiments
The first experiment is a controlled synthetic study, which is where the claims can be separated cleanly. Families of $M$ stable systems are generated in the style of the Autom paper, with $A_m$ in a $k$-dimensional span. On top of that design we vary three things: the spread of trajectory lengths across systems, from uniform to heavily skewed; the fraction of systems given an off-span deviation of increasing size; and how poorly each trajectory excites individual state directions. The metrics are normalized Frobenius error $\|\hat A_m - A_m\|_F / \|A_m\|_F$, reported as median and worst-system values, along with the fraction of systems whose error exceeds their individual least-squares error. That last quantity is a direct count of negative transfer, playing the role that $\Delta m\%$ plays in deep multi-task benchmarks. The decisive check here is twofold. Our method should be the only one whose worst-system error stays at or below individual least squares across the full off-span sweep. It should also match the factorized method when every system lies in the span.
The second experiment uses the HCP fMRI setting of Multi-Task System Identification of Similar Linear Time-Invariant Dynamical Systems, with subjects as systems. We subsample scans unevenly across subjects to create a realistic quality spread. Evaluation is one-step and multi-step prediction error on held-out scans. The decisive check is whether data-limited subjects improve while well-scanned subjects are not degraded relative to their own least-squares fits. Both convex baselines and gradient balancing are expected to trade one group against the other.
The third experiment is a downstream test on the Dynamic Texture Database used in Probabilistic Kernels for the Classification of Auto-Regressive Visual Processes, with each video treated as a system. To put all transition matrices in comparable state coordinates, we fix one appearance matrix $C$ per database by principal component analysis over all training videos. We then truncate clips to a small fraction of their length, identify $A$ for each video by each method, and classify with the KL-divergence kernel support vector machine from that paper. That paper reports 96% accuracy with full-length clips and 89% for the Martin-distance baseline. These figures are a ceiling and a reference, not a direct comparison, since our clips are truncated. The decisive check is how quickly accuracy degrades as clips shorten. If joint identification keeps accuracy closer to 96% than individual identification does at the same truncation, sharing has preserved the dynamics information that classification needs. If the scalar-weight ablation keeps up, direction-wise borrowing adds nothing in this application.
References
- Multi-Task System Identification of Similar Linear Time-Invariant Dynamical Systems
- Joint Learning of Linear Time-Invariant Dynamical Systems
- Multi-task imitation learning for linear dynamical systems
- Conflict-Averse Gradient Descent for Multi-task learning
- RotoGrad: Gradient Homogenization in Multi-Task Learning
- Probabilistic Kernels for the Classification of Auto-Regressive Visual Processes