A computational model of hippocampal function in trace conditioning

Elliot A. Ludvig, Richard S. Sutton, Eric Verbeek, E. James KehoeNeurIPS 2008 · OpenReview

Researchers have developed a reinforcement learning model that explains a long-standing neuroscientific puzzle: why the hippocampus is essential for "trace" conditioning (learning with a time gap) but often unnecessary for "delay" conditioning (learning with overlapping stimuli). By representing stimuli as a collection of decaying "temporal microstimuli," the model demonstrates that hippocampal damage effectively "shortens" the brain's internal clock, making it impossible to bridge the temporal gap required for trace conditioning.

The Challenge: Trace vs. Delay Conditioning

In classical conditioning, an animal learns to associate a conditioned stimulus (CS, like a tone) with an unconditioned stimulus (US, like a puff of air to the eye). Two primary paradigms are used to study this:

  • Delay Conditioning: The tone stays on until the air puff occurs. The stimuli overlap.
  • Trace Conditioning: The tone ends, followed by a silent "trace interval" before the air puff.
Trace vs. Delay Conditioning Paradigms

Neuroscientists have consistently observed that animals with hippocampal damage can learn delay conditioning without issue, but they fail entirely at trace conditioning. This paper seeks to provide a unified computational explanation for this phenomenon. The authors argue that the hippocampus isn't just a generic "memory box," but a provider of specific temporal building blocks that allow the brain to track associations across time.

Methods: Modeling Temporal Microstimuli

The authors utilize a Temporal-Difference (TD) learning model, a foundational algorithm in reinforcement learning. The core innovation of their approach lies in the Stimulus Representation. Instead of treating a stimulus as a single monolithic event, the model breaks it down into two components:

  1. Presence Microstimuli: A simple, persistent signal that remains "on" only while the external stimulus is physically present.
  2. Temporal Microstimuli: A series of internal signals triggered by the onset of a stimulus. These signals peak at different delays and gradually broaden over time. They are created via coarse coding of a decaying memory trace using Gaussian basis functions.

Simulating Hippocampal Lesions: The authors model hippocampal damage by increasing the decay rate of the underlying memory trace. This effectively "compresses" the temporal microstimuli, eliminating the long-latency elements that would normally persist during a long gap. The model then uses a linear TD($\lambda$) algorithm to update weights and a "leaky-integrator" rule to translate these internal predictions into observable behavior (conditioned responses).

Key Insights: The Timing Problem

The model reveals that the hippocampus's role in trace conditioning is fundamentally about timing rather than simple memory storage.

  • The "Presence" Advantage: In delay conditioning, the "presence microstimulus" is available right up until the reward. This makes the timing problem easy; the brain doesn't have to rely on fading internal traces to know when the reward is coming.
  • The Hippocampus as a Temporal Bridge: In trace conditioning, the "presence" signal disappears before the reward arrives. The brain must rely on long-latency temporal microstimuli to bridge the gap. If the hippocampus is responsible for maintaining these long-duration traces, its removal specifically cripples trace learning.

Key Results: Replicating Biological Realities

To test the model, the authors simulated experimental protocols based on classic behavioral studies, comparing Normal vs. Hippocampal-lesioned states across Short (250ms), Medium (500ms), and Long (1000ms) Inter-Stimulus Intervals (ISI).

Simulated Learning Curves across ISIs
  • Selective Impairment: The model successfully replicated empirical data. Simulated hippocampal damage severely impaired trace conditioning at medium (500ms) and long (1000ms) intervals, while leaving delay conditioning largely intact.
  • Interval Sensitivity: At very short intervals (250ms), the model showed that even lesioned "brains" could learn trace conditioning. This matches biological observations that the hippocampus is less critical when the temporal gap is minimal enough for short-latency traces to survive.
  • Response Topography: The model demonstrated that in normal trace conditioning, the learned response peaks right at the expected time of the reward. In lesioned simulations that managed to learn (at short intervals), the responses tended to occur earlier—a subtle shift that serves as a testable prediction for future animal studies.

Limitations and Future Work

While powerful, the model uses a simplified, abstract Gaussian basis function approach. The authors acknowledge that incorporating more biologically detailed mechanisms—such as specific hippocampal subfields or theta oscillations—would be a natural next step. Furthermore, the specific biological reason why hippocampal damage causes faster trace decay is assumed rather than explicitly modeled. Finally, while this model beautifully explains aversive conditioning (like eyeblink responses), extending this TD framework to encompass complex spatial navigation or episodic memory remains an exciting frontier for future research.

Summary updated

Abstract

We present a new reinforcement-learning model for the role of the hippocampus in classical conditioning, focusing on the differences between trace and delay conditioning. In the model, all stimuli are represented both as unindividuated wholes and as a series of temporal elements with varying delays. These two stimulus representations interact, producing different patterns of learning in trace and delay conditioning. The model proposes that hippocampal lesions eliminate long-latency temporal elements, but preserve short-latency temporal elements. For trace conditioning, with no contiguity between stimulus and reward, these long-latency temporal elements are vital to learning adaptively timed responses. For delay conditioning, in contrast, the continued presence of the stimulus supports conditioned responding, and the short-latency elements suppress responding early in the stimulus. In accord with the empirical data, simulated hippocampal damage impairs trace conditioning, but not delay conditioning, at medium-length intervals. With longer intervals, learning is impaired in both procedures, and, with shorter intervals, in neither. In addition, the model makes novel predictions about the response topography with extended stimuli or post-training lesions. These results demonstrate how temporal contiguity, as in delay conditioning, changes the timing problem faced by animals, rendering it both easier and less susceptible to disruption by hippocampal lesions.
BibTeX
@inproceedings{ludvig2008computational,
  author = {Elliot A. Ludvig and Richard S. Sutton and Eric Verbeek and E. James Kehoe},
  title = {{A computational model of hippocampal function in trace conditioning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2008},
  url = {https://openreview.net/forum?id=B1ZX08-ObB},
}

Loading related content...

Figure