Rules and mechanisms for efficient two-stage learning in neural circuits

Tiberiu Teşileanu, Bence P. Ölveczky, Vijay BalasubramanianeLife 2017 · DOI

TL;DR

This paper introduces a computational model of two-stage learning, inspired by how songbirds learn their songs. They show that efficient learning hinges on a precise temporal match between a "tutor" circuit's teaching signal and a "student" circuit's plasticity rules. This framework can predict how neural circuits should communicate to learn complex behaviors and is applicable beyond birdsong.

What Problem Does This Paper Address?

Many learning processes in the brain involve a "two-stage" system. For instance, in songbirds, a circuit called LMAN (part of the basal ganglia) generates exploratory vocalizations, while another circuit, RA (a motor cortex analog), consolidates these into a stable song. Prior models often treated LMAN as a random noise generator. However, recent evidence suggests LMAN provides a more structured, corrective bias. This paper addresses how this "tutor" signal from LMAN should be structured to effectively guide the "student" circuit (RA) in learning complex, temporally patterned outputs like birdsong, especially when the tutor doesn't have a perfect model of the student.

Methods

The authors developed a computational model that captures this two-stage learning architecture. They used a rate-based model for analytical tractability, which was then extended to a spiking neuron model to better reflect biological realism. A key innovation is their derivation of an "optimal tutor signal" using stochastic gradient descent. This signal is designed to match the student's specific plasticity rules, ensuring efficient learning. They also explored how a tutor could learn this signal through reinforcement learning, even without direct knowledge of the student's internal workings.

Datasets

Not discussed in the paper. The authors used simulated data generated by their models.

Key Insights and Ideas

  • Tutor-Student Temporal Matching is Crucial: The most surprising finding is the critical importance of the temporal alignment between the tutor's teaching signal and the student's plasticity mechanism. Mismatches, especially in the timescale of error integration, can severely impair or even halt learning.
  • Beyond Random Noise: LMAN's role isn't just to inject random variability; it provides a structured, corrective bias that must be carefully timed to guide learning.
  • Reinforcement Learning for Tutor Signal Generation: The tutor can learn to generate an effective teaching signal through reinforcement, even if it doesn't fully understand the student's internal learning rules. This is a more biologically plausible scenario.
  • Predictive Power for Neural Circuits: The model provides concrete, testable predictions about the temporal dynamics of tutor signals (like LMAN activity) based on the student's plasticity rules (like those in RA).

Key Results

  • Efficient Learning with Matched Tutors: Simulations showed that when the tutor's error integration timescale matched the student's plasticity rule, learning was rapid and accurate.
  • Impaired Learning with Mismatched Tutors: Mismatches in the tutor's timescale led to significantly slower learning or complete failure. Specifically, a tutor timescale shorter than optimal disrupted learning, while a longer timescale slowed it down.
  • Spiking Neuron Model Reproduces Biological Patterns: The spiking neuron model successfully reproduced key statistical features of RA neuron activity observed in juvenile and adult songbirds as they learn.
  • Robustness to Nonlinearities and Errors: The model demonstrated robustness to nonlinearities in neuron activation and output, and even to some degree of credit assignment error (where the tutor incorrectly attributes outcomes to specific student actions).

Limitations and Future Work

  • Simplifications in the Model: The authors acknowledge that their model simplifies certain aspects, such as the complex connectivity of real neural circuits and the precise biochemical mechanisms of plasticity. They also note that their model of HVC neurons firing single bursts might be an oversimplification.
  • Future Directions:
    • Investigating the role of recurrent circuitry in LMAN for implementing the integral tutor signal.
  • Extending the model to hierarchical learning systems with more than two stages.
  • Exploring the impact of nonlinearities in student neuron activation and output functions more deeply.
  • Testing the model's predictions in experimental settings by manipulating tutor signals and measuring student plasticity.
  • Incorporating more detailed models of synaptic plasticity, including NMDA receptor dynamics and developmental changes.

Summary updated

Abstract

Trial-and-error learning requires evaluating variable actions and reinforcing successful variants. In songbirds, vocal exploration is induced by LMAN, the output of a basal ganglia-related circuit that also contributes a corrective bias to the vocal output. This bias is gradually consolidated in RA, a motor cortex analogue downstream of LMAN. We develop a new model of such two-stage learning. Using stochastic gradient descent, we derive how the activity in 'tutor' circuits (<i>e.g.,</i> LMAN) should match plasticity mechanisms in 'student' circuits (<i>e.g.,</i> RA) to achieve efficient learning. We further describe a reinforcement learning framework through which the tutor can build its teaching signal. We show that mismatches between the tutor signal and the plasticity mechanism can impair learning. Applied to birdsong, our results predict the temporal structure of the corrective bias from LMAN given a plasticity rule in RA. Our framework can be applied predictively to other paired brain areas showing two-stage learning.
BibTeX
@inproceedings{tesileanu2017rules,
  author = {Tiberiu Teşileanu and Bence P. Ölveczky and Vijay Balasubramanian},
  title = {{Rules and mechanisms for efficient two-stage learning in neural circuits}},
  booktitle = {eLife},
  year = {2017},
  doi = {10.7554/elife.20944},
}
Figure