Efficient learning of nonlinear prediction models with time-series privileged information

Bastian Jung, Fredrik Daniel Johansson

Introduction

In many real-world machine learning scenarios, we have access to "extra" data during training that will not be available when the model is deployed. For example, when predicting a patient's health outcome six months after their first visit, training data might include the full history of their lab tests and vitals over those six months. However, during deployment, only the initial exam is available.

Standard supervised learning often ignores this intermediate "privileged" information. While previous research demonstrated that utilizing this data improves sample efficiency in simple linear systems, applying it to the complex, nonlinear dynamics typical of real-world data remained an open challenge. This paper extends the "Learning using Privileged Information" (LuPI) framework to nonlinear time-series prediction, proving that access to intermediate data during training consistently reduces prediction risk and improves sample efficiency.

Graphical models comparing fully observed linear systems to latent dynamics with nonlinear observations.

Bridging the Gap to Nonlinear Reality

To leverage privileged information in complex environments, the authors propose several methodological advances:

  1. Generalized LuPTS: The authors extend a linear algorithm (LuPTS) to handle nonlinearities. They provide theoretical guarantees showing that if the underlying system has linear latent dynamics, a privileged learner is "never worse" than a classical learner in expectation.
  2. Random Feature Maps: To manage unknown nonlinearities without sacrificing theoretical guarantees, the framework employs Random Fourier Features (RFF) and Random ReLU Features (RRF). These project inputs into a high-dimensional space where linear methods can be applied effectively.
  3. Privileged Representation Learning: For deep learning contexts, the authors introduce three distinct architectures to capture intermediate dynamics:
  4. Stepwise (SRL): Learns to predict the next latent state in the sequence.
  5. Greedy (GRL): Learns to predict the final outcome from every intermediate time step.
  6. Combined (CRL): A hybrid approach balancing next-step prediction with final-outcome prediction.

Key Insights and Empirical Results

The proposed models were evaluated across a diverse set of synthetic and real-world datasets, including interstate traffic volume, Alzheimer’s disease progression (ADNI), and complex image sequences (Clocks-LGS).

Variance Reduction vs. Bias Amplification

The primary mechanism by which privileged information aids learning is variance reduction. By supervising the model at every intermediate step, the learning process becomes significantly more stable, which is critical in low-data regimes. However, the authors identify a "bias trap": if the model architecture is a poor fit for the data, privileged information can actually amplify bias because errors at each time step accumulate.

Superior Sample Efficiency and Latent Discovery

Across almost all tasks, the nonlinear LuPTS variants outperformed classical Ordinary Least Squares (OLS) and standard neural networks, particularly when training samples were scarce. Furthermore, privileged learners proved significantly better at recovering the true underlying states of a system. Using SVCCA (Singular Vector Canonical Correlation Analysis), the authors demonstrated that their models' internal representations correlated much more highly with true latent variables than those of standard models. In image-based tasks, the Greedy (GRL) and Combined (CRL) representation learners achieved dominant $R^2$ scores and superior latent recovery.

The Overparameterization Limit

Interestingly, the advantage of privileged information vanishes when the number of model features significantly exceeds the number of training samples—a phenomenon observed in the "double descent" regime.

Limitations and Future Work

While the empirical results demonstrate robustness across various data modalities, the theoretical "never worse" guarantee currently relies on the assumption of a latent linear-Gaussian system. A more general theory for purely nonlinear latent systems remains an open problem. Additionally, mitigating the bias amplification issue is a crucial next step. Future work will likely focus on developing risk guarantees for biased representations and extending the LuPI framework to more complex causal structures, such as general Directed Acyclic Graphs (DAGs).

Summary updated

Abstract

In domains where sample sizes are limited, efficient learning algorithms are critical. Learning using privileged information (LuPI) offers increased sample efficiency by allowing prediction models access to auxiliary information at training time which is unavailable when the models are used. In recent work, it was shown that for prediction in linear-Gaussian dynamical systems, a LuPI learner with access to intermediate time series data is never worse and often better in expectation than any unbiased classical learner. We provide new insights into this analysis and generalize it to nonlinear prediction tasks in latent dynamical systems, extending theoretical guarantees to the case where the map connecting latent variables and observations is known up to a linear transform. In addition, we propose algorithms based on random features and representation learning for the case when this map is unknown. A suite of empirical results confirm theoretical findings and show the potential of using privileged time-series information in nonlinear prediction.
BibTeX
@inproceedings{jung2022efficient,
  author = {Bastian Jung and Fredrik Daniel Johansson},
  title = {{Efficient learning of nonlinear prediction models with time-series privileged information}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=V9ngeCMsZK3},
}
Figure