An Online Algorithm for Learning Selectivity to Mixture Means

Matthew Lawlor, Steven W. ZuckerarXiv 2014 · arXiv

Abstract

A fundamental challenge in both machine learning and neuroscience is "selectivity"—the ability of a system to learn and respond to specific patterns within noisy data streams. While standard methods like Expectation-Maximization (EM) lack global convergence and online capabilities, classical biological rules like BCM often fail with complex distributions. This paper introduces Triplet BCM, an online learning rule that leverages "triplets" of data samples to provably learn the means of mixture models. By connecting neural plasticity to spectral tensor methods, we provide a robust information-processing interpretation for spike-timing-dependent plasticity (STDP).

Introduction

Learning to categorize inputs from a continuous stream is essential for biological survival and artificial intelligence. However, existing algorithms often face a trade-off between biological plausibility and mathematical robustness. The Bienenstock, Cooper, and Munro (BCM) rule is a well-known biologically grounded model for synaptic plasticity, yet it struggles to converge in high-noise environments or with complex data structures.

The authors address this by introducing a "multi-view" assumption: the environment is assumed to provide at least three samples (a triplet) from the same hidden cause before that cause changes. This allows the formulation of a learning rule that is both online—learning one sample at a time—and mathematically guaranteed to find the true underlying means of a mixture distribution.

The Triplet BCM Rule

The core innovation of Triplet BCM is its interpretation of neural learning as a form of tensor decomposition. By processing data in triplets, the rule constructs third-order moment tensors that are significantly more resilient to noise than single-sample methods.

The learning rule is derived as a stochastic gradient ascent on a specific objective function. A key component of this process is the sliding threshold, which regulates the neuron's selectivity.

The BCM objective function $\phi(c, \theta)$ determines the change in synaptic weights based on the output $c$ relative to a sliding threshold $\theta$.

As shown in the figure above, the function $\phi(c, \theta)$ dictates how weights are adjusted. When the output $c$ is below the threshold $\theta$, the weights are decreased (long-term depression); when $c$ exceeds $\theta$, weights are increased (long-term potentiation). The threshold $\theta$ itself evolves based on the history of the neuron's firing rate, ensuring the neuron remains selective to specific features of the input distribution.

To enable a population of neurons to learn different components of a mixture, the authors incorporate lateral inhibition. A lateral connection matrix ($L$) allows neurons to perform a parallel search, preventing them from all converging to the same mixture mean.

Results and Validation

The paper validates the Triplet BCM rule through both mathematical proof and synthetic experiments.

Provable Convergence

Using stochastic approximation theory, the authors prove that the Triplet BCM rule converges with probability one to the true means of a mixture model, provided these means are linearly independent. This provides a rigorous foundation for a model that has been studied in neuroscience for decades.

Synthetic Mixture Models

The theory was tested using a 2D Gaussian Mixture Model with two components. The experiments demonstrated that:

  • Selectivity: Each neuron successfully becomes selective to exactly one mixture component.
  • Lateral Control: The sign of lateral connections determines population behavior. In an inhibitory network, neurons learned different components 83.3% of the time. Conversely, excitatory connections forced neurons to synchronize and learn the same component.

Discussion and Insights

The Triplet BCM framework offers a new perspective on several neuroscientific phenomena:

  • Spike Trains: The requirement for "multiple views" suggests a functional reason for why neurons fire in bursts or trains; disjoint spiking intervals may provide the necessary temporal samples for robust tensor-based learning.
  • Noise Robustness: By utilizing the conditional independence of samples within a triplet, the rule effectively "cancels out" noise that is not shared across the views, a significant improvement over classical BCM.

While the rule currently requires that the number of mixture components does not exceed the input dimensionality, it represents a major step toward biologically plausible, mathematically robust online learning. Future work aims to extend this to overcomplete representations and further formalize the link to specific biological mechanisms like STDP.

Summary updated

BibTeX
@misc{lawlor2014online,
  author = {Matthew Lawlor and Steven W. Zucker},
  title = {{An Online Algorithm for Learning Selectivity to Mixture Means}},
  year = {2014},
  eprint = {1410.8580},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1410.8580},
}
Figure