Training Dynamic Exponential Family Models with Causal and Lateral Dependencies for Generalized Neuromorphic Computing

Hyeryung Jang, Osvaldo Simeone

TL;DR

This paper introduces a novel probabilistic model, the "dynamic exponential family," designed to capture both causal and instantaneous dependencies in time-series data, making it suitable for neuromorphic computing. The authors develop distributed learning algorithms for this model, demonstrating its effectiveness on a multi-task supervised learning problem.

What Problem Does This Paper Address?

Traditional Artificial Neural Networks (ANNs) are computationally intensive. Neuromorphic computing, inspired by the brain, offers a more energy-efficient alternative using Spiking Neural Networks (SNNs). However, training SNNs, especially with probabilistic models that can capture complex temporal dynamics and correlations, remains a significant challenge. Existing probabilistic models for SNNs often struggle with arbitrary data alphabets and fail to fully account for both causal (over time) and lateral (instantaneous) dependencies between neurons. This paper aims to bridge this gap by proposing a flexible probabilistic framework and efficient learning methods for SNNs that can handle these complexities.

Methods

The core contribution is the dynamic exponential family model. This model extends exponential family harmoniums to time series by using a hybrid directed-undirected graphical representation:

  • Causal Graph ($G_p$): Represents directed causal relationships between time series (neurons) over time.
  • Lateral Graph ($G_c$): Represents instantaneous correlations (lateral connections) between neurons at the same time step.

The model's conditional probability distribution is defined using sufficient statistics and an energy function that incorporates causal and lateral dependencies. To learn the model parameters, the authors derive distributed learning rules for both:

  • Maximum Likelihood (ML): Using gradient descent, where each unit can compute its gradients locally using information from its causal and lateral neighbors.
  • Bayesian Learning: Employing stochastic gradient Langevin dynamics, which allows for capturing parameter uncertainty and avoiding overfitting.

A key aspect of the learning is the use of basis functions (e.g., raised cosines) to parameterize the causal dependencies, reducing the number of parameters and enabling efficient computation of spatio-temporal receptive fields.

Datasets

The paper evaluates its methods on a multi-task supervised learning problem using the USPS handwritten digit dataset. Specifically, they use 500 images of digits "1" and "7" for training and 125 for testing. The images are used in both their original and rotated forms to create two classification tasks: digit identification and orientation detection.

Key Insights and Ideas

  • Unified Probabilistic Framework for SNNs: The dynamic exponential family model provides a principled way to model SNNs that accounts for both temporal (causal) and instantaneous (lateral) interactions, moving beyond simpler deterministic or purely causal models.
  • Distributed Learning for Neuromorphic Hardware: The derived ML and Bayesian learning rules are designed to be distributed, meaning each neuron can compute its updates using local information. This is crucial for efficient implementation on neuromorphic hardware where communication between units is a bottleneck.
  • Lateral Connections Improve Performance: The experiments show that incorporating lateral connections between output neurons significantly improves classification accuracy for both tasks, highlighting the importance of modeling instantaneous correlations for tasks involving related outputs.
  • Bayesian Priors for Regularization: The Bayesian learning approach demonstrates that the choice of prior distribution for model parameters can effectively combat overfitting, with a bimodal prior showing superior generalization performance compared to a uniform prior.

Key Results

  • The proposed dynamic exponential family model, trained with ML, achieves improved classification accuracy on digit and orientation detection tasks when lateral connections are utilized.
  • For the digit classification task, accuracies with lateral connections are consistently higher across different spike train lengths (T). For orientation classification, the benefit of lateral connections is even more pronounced.
  • The Bayesian learning approach, particularly with a bimodal prior, shows good generalization performance and avoids overfitting, as evidenced by the log-likelihood curves and parameter evolution plots.

Limitations and Future Work

  • Fully Observed Time Series: The current work focuses on fully observed time series. The authors mention that extending the framework to handle latent variables (unobserved states) is a direction for future work.
  • Computational Complexity: For large lateral graphs ($G_c$), the marginalization steps in gradient computation can become computationally expensive. The paper suggests using techniques like Gibbs sampling or Contrastive Divergence to address this.
  • Generalization to More Complex SNNs: While applied to a two-layer SNN, the framework is general and could be extended to deeper and more complex SNN architectures.

Summary updated

Abstract

Neuromorphic hardware platforms, such as Intel's Loihi chip, support the implementation of Spiking Neural Networks (SNNs) as an energy-efficient alternative to Artificial Neural Networks (ANNs). SNNs are networks of neurons with internal analogue dynamics that communicate by means of binary time series. In this work, a probabilistic model is introduced for a generalized set-up in which the synaptic time series can take values in an arbitrary alphabet and are characterized by both causal and instantaneous statistical dependencies. The model, which can be considered as an extension of exponential family harmoniums to time series, is introduced by means of a hybrid directed-undirected graphical representation. Furthermore, distributed learning rules are derived for Maximum Likelihood and Bayesian criteria under the assumption of fully observed time series in the training set.
BibTeX
@inproceedings{jang2019training,
  author = {Hyeryung Jang and Osvaldo Simeone},
  title = {{Training Dynamic Exponential Family Models with Causal and Lateral Dependencies for Generalized Neuromorphic Computing}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  doi = {10.1109/icassp.2019.8682960},
}
Figure