Learning Unstable Dynamical Systems with Time-Weighted Logarithmic Loss

Kamil Nar, Yuan Xue, Andrew M. DaiarXiv 2020 · OpenReview

TL;DR

Training linear dynamical systems, especially unstable ones, with standard gradient descent and squared-error loss often fails to converge. This paper introduces a novel "time-weighted logarithmic loss" function that effectively balances the influence of data points across time, enabling stable and accurate learning of unstable dynamical systems.

What Problem Does This Paper Address?

System identification, the task of learning the parameters of a dynamical model from observed data, is crucial for many applications like reinforcement learning, medical record analysis, and financial forecasting. A key challenge arises when dealing with unstable dynamical systems. Traditional methods using gradient descent with squared-error loss struggle to converge when the system's dynamics are unstable. While restricting the parameter space can help for stable systems, it's insufficient for unstable ones. This paper investigates why gradient descent fails for unstable systems and proposes a new loss function to overcome this limitation. The problem is further complicated by irregularly sampled data, which often requires imputation or specialized models.

Methods

The core of this paper lies in analyzing the convergence dynamics of gradient descent for linear dynamical systems. The authors first derive an upper bound on the learning rate for the squared-error loss, showing that it becomes prohibitively small for unstable systems. This bound reveals that data points sampled at different times have vastly different impacts on the learning process, with later samples imposing exponentially stricter constraints.

To address this imbalance, they introduce a time-weighted logarithmic loss function. This new loss function is designed to "balance the influence of all samples on the dynamics of the algorithm," making it easier for gradient descent to converge to the global minimum, even for unstable systems. The paper provides theoretical guarantees in the form of necessary conditions for convergence for both the squared-error and the proposed time-weighted logarithmic loss.

Datasets

The paper does not introduce new datasets. Instead, it relies on synthetic data generated for experimental evaluation. The authors generated output trajectories from randomly generated linear systems (both stable and unstable) with varying initial states and observation matrices. These synthetic datasets were used to train and evaluate the performance of their proposed loss function against the standard mean-squared-error loss.

Key Insights and Ideas

  • Time-dependent data influence: The key insight is that for unstable systems, the temporal ordering of data points significantly skews the gradient descent optimization landscape. Later data points, especially from an unstable system, can dominate the loss and make convergence impossible with standard loss functions.
  • The "unstable mode" problem: The squared-error loss struggles to learn all modes of an unstable system. It might correctly identify the unstable mode but fail to capture stable or marginally stable modes, leading to an incomplete and inaccurate model.
  • Logarithmic loss for balance: The proposed time-weighted logarithmic loss acts as a "regularizer" across time, preventing any single time point or group of points from disproportionately influencing the gradient. This "balances the weights" of all samples, leading to more robust convergence.
  • Theoretical underpinnings: The paper provides theoretical bounds on learning rates and convergence conditions, offering a rigorous justification for why the proposed loss function is effective.

Key Results

  • Experimental validation: Experiments show that the time-weighted logarithmic loss function allows gradient descent to effectively decrease the training error for unstable systems, whereas the mean-squared-error loss fails to do so.
  • Accurate mode discovery: Figure 2 demonstrates that when learning an unstable three-dimensional system, the mean-squared-error loss only correctly identifies the unstable mode. In contrast, the time-weighted logarithmic loss successfully discovers all three modes of the system. Similar results are shown for a four-dimensional system in Figure 4.
  • Theoretical convergence: The paper provides theoretical theorems that establish necessary conditions for gradient descent convergence with both loss functions, highlighting the significant relaxation offered by the time-weighted logarithmic loss for unstable systems.

Limitations and Future Work

  • Linear systems focus: The current work is focused on linear dynamical systems. The authors suggest that an important future direction is to investigate whether a similar loss function can improve the training of nonlinear models.
  • Deterministic framework: The analysis is primarily conducted within a deterministic framework. The paper raises the question of whether allowing randomness in the system's state or transitions could offer a trade-off between model accuracy and training efficiency.
  • Initial state handling: While the paper considers different scenarios for initial states (known, learned, or estimated), the theoretical results are most directly applicable when the estimation error at equilibrium is zero.
  • Synthetic data: The experiments are conducted on synthetic data. While illustrative, testing on real-world, irregularly sampled datasets would be valuable.

Summary updated

Abstract

When training the parameters of a linear dynamical model, the gradient descent algorithm is likely to fail to converge if the squared-error loss is used as the training loss function. Restricting the parameter space to a smaller subset and running the gradient descent algorithm within this subset can allow learning stable dynamical systems, but this strategy does not work for unstable systems. In this work, we look into the dynamics of the gradient descent algorithm and pinpoint what causes the difficulty of learning unstable systems. We show that observations taken at different times from the system to be learned influence the dynamics of the gradient descent algorithm in substantially different degrees. We introduce a time-weighted logarithmic loss function to fix this imbalance and demonstrate its effectiveness in learning unstable systems.
BibTeX
@misc{nar2020learning,
  author = {Kamil Nar and Yuan Xue and Andrew M. Dai},
  title = {{Learning Unstable Dynamical Systems with Time-Weighted Logarithmic Loss}},
  year = {2020},
  eprint = {2007.05189},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.05189},
}
Figure