Universal Approximation and Learning of Trajectories Using Oscillators

Pierre Baldi, Kurt HornikCaltechAUTHORS 1995

TL;DR

This paper proposes a novel approach to learning complex trajectories for neural networks by combining simple oscillators in a modular, hierarchical architecture. They demonstrate that such "oscillator banks" possess universal approximation properties, meaning they can, in principle, generate any desired trajectory, overcoming limitations of traditional gradient descent methods.

What Problem Does This Paper Address?

Designing artificial neural networks, particularly in robotics, often requires them to generate specific, dynamic paths or "trajectories" in their state space. Think of a robot arm needing to trace a precise curve. While learning these trajectories from examples seems intuitive, standard gradient descent methods struggle with complex, time-varying patterns in "amorphous" (unstructured) neural networks. This difficulty arises from issues like abrupt changes in the network's behavior (bifurcations) that disrupt the learning process. Biological systems, like central pattern generators, also face similar challenges in producing coordinated movements. The paper aims to find a more robust and structured way for neural networks to learn and generate these dynamic trajectories.

Methods

The authors propose an "oscillator approach" that moves away from amorphous networks. Instead, they advocate for building networks from "weakly pre-structured" modules, specifically simple oscillators. These oscillators are combined in a hierarchical, modular fashion. The paper contrasts two main ideas:

  1. Modular Hierarchies: Complex trajectories are built by combining simpler oscillatory modules. For example, a double figure-eight trajectory could be constructed by first creating modules for individual loops, then combining these into modules for figure eights, and finally assembling these into the full double figure-eight. This creates a more organized structure.
  2. Universal Approximation with Oscillators: The core theoretical contribution is demonstrating that banks of oscillators, even with bounded frequencies, can universally approximate any continuous trajectory. This means that with enough oscillators and appropriate tuning, they can generate virtually any desired path. They show this is possible both by dynamically switching between oscillators (Equation 3) and by using linear combinations of oscillators with constant coefficients (Equations 4 and Theorem 1).

Datasets

Not discussed in the paper. The paper focuses on theoretical demonstrations and conceptual examples (like the double figure-eight trajectory) rather than empirical evaluation on specific datasets.

Key Insights and Ideas

  • Structure is Key for Trajectory Learning: The paper strongly suggests that unstructured, "amorphous" neural networks are inherently ill-suited for learning complex trajectories. Introducing modularity and structure, specifically through oscillators, is crucial.
  • Oscillators as Building Blocks: Simple oscillators, when combined hierarchically, can serve as powerful building blocks for generating dynamic behaviors. This is inspired by biological systems.
  • Bounded Frequencies are Sufficient: A significant theoretical result is that you don't need infinitely many or arbitrarily high-frequency oscillators to achieve universal approximation. A bounded range of frequencies is enough, which is more biologically plausible and potentially more efficient.
  • Trade-off Between Frequency and Coefficients: There's an inherent trade-off: using a wider range of frequencies might require smaller mixing coefficients, while a narrower band might necessitate larger coefficients.

Key Results

The paper's primary result is the theoretical demonstration that architectures composed of banks of oscillators possess universal approximation properties for trajectories. This means they can approximate any continuous function (trajectory) over a given time interval to an arbitrary degree of accuracy. They present theorems (Theorem 1, 2, and 3) that formalize this capability, showing that such approximations are possible using oscillators with frequencies within a bounded interval, or even with a fixed set of frequencies.

Limitations and Future Work

The authors acknowledge several open questions and areas for future research:

  • Learning Across Modules: How to effectively coordinate learning across different hierarchical levels and modules remains a fundamental challenge.
  • Origin of Decomposition: Can the modular architecture emerge organically from a learning process, rather than being externally imposed?
  • Global Gradient Descent: How successful would a global gradient descent strategy be when applied across these modular architectures?
  • Flexibility and Switching: How can the same modular architecture be used for different trajectories, and how can switching between them be achieved quickly with proper phase adjustments?
  • Practical Implementation: The paper hints at a fundamental trade-off between the size of frequencies and the size of mixing coefficients, which is still under investigation for practical implementation.

Summary updated

Abstract

Natural and artificial neural circuits must be capable of traversing specific state space trajectories. A natural approach to this problem is to learn the relevant trajectories from examples. Unfortunately, gradient descent learning of complex trajectories in amorphous networks is unsuccessful. We suggest a possible approach where trajectories are realized by combining simple oscillators,
\nin various modular ways. We contrast two regimes of fast
\nand slow oscillations. In all cases, we show that banks of oscillators with bounded frequencies have universal approximation properties. Open questions are also discussed briefly.
BibTeX
@inproceedings{baldi1995universal,
  author = {Pierre Baldi and Kurt Hornik},
  title = {{Universal Approximation and Learning of Trajectories Using Oscillators}},
  booktitle = {CaltechAUTHORS},
  year = {1995},
}
Figure