Accounting for Negative Automaintenance in Pigeons: A Dual Learning Systems Approach and Factored Representations

Florian Lesaint, Olivier Sigaud, Mehdi KhamassiPLoS 2014 · DOI

Abstract

Animals, including Humans, are prone to develop persistent maladaptive and suboptimal behaviours. Some of these behaviours have been suggested to arise from interactions between brain systems of Pavlovian conditioning, the acquisition of responses to initially neutral stimuli previously paired with rewards, and instrumental conditioning, the acquisition of active behaviours leading to rewards. However the mechanics of these systems and their interactions are still unclear. While extensively studied independently, few models have been developed to account for these interactions. On some experiment, pigeons have been observed to display a maladaptive behaviour that some suggest to involve conflicts between Pavlovian and instrumental conditioning. In a procedure referred as negative automaintenance, a key light is paired with the subsequent delivery of food, however any peck towards the key light results in the omission of the reward. Studies showed that in such procedure some pigeons persisted in pecking to a substantial level despite its negative consequence, while others learned to refrain from pecking and maximized their cumulative rewards. Furthermore, the pigeons that were unable to refrain from pecking could nevertheless shift their pecks towards a harmless alternative key light. We confronted a computational model that combines dual-learning systems and factored representations, recently developed to account for sign-tracking and goal-tracking behaviours in rats, to these negative automaintenance experimental data. We show that it can explain the variability of the observed behaviours and the capacity of alternative key lights to distract pigeons from their detrimental behaviours. These results confirm the proposed model as an interesting tool to reproduce experiments that could involve interactions between Pavlovian and instrumental conditioning. The model allows us to draw predictions that may be experimentally verified, which could help further investigate the neural mechanisms underlying theses interactions.

Introduction

In classical conditioning, animals learn that a stimulus (like a light) predicts a reward (food). In instrumental conditioning, they learn that specific actions lead to rewards. Usually, these systems work in harmony. However, "negative automaintenance" reveals a glitch: if a light predicts food, but pecking that light cancels the food, many pigeons cannot stop themselves from pecking. This paper proposes a computational model to explain this puzzling behavior—a maladaptive persistence where pigeons lose their food reward by interacting with the very cue that predicts it.

The authors address two main gaps in current behavioral literature: the significant individual variability between pigeons (some learn to refrain, others do not) and the need for a unified model that explains both this failure and other behaviors like "sign-tracking" and "goal-tracking" seen in rodents.

The Dual-Learning Model

The core of the proposed model consists of two interacting reinforcement learning (RL) systems that process environmental information differently:

  • Model-Based (MB) System: This is the "rational" planner. It learns a map of the environment (transitions and rewards) and calculates the "advantage" of an action based on long-term reward maximization. It "knows" that pecking leads to reward omission.
  • Feature-Model-Free (FMF) System: This is the "impulsive" or Pavlovian system. Instead of abstract states, it learns values associated with specific factored representations (features like "key light" or "food magazine"). It assigns high "incentive salience" to any feature that predicts reward, regardless of the consequences of interacting with it.
The dual-learning architecture combining Model-Based (MB) and Feature-Model-Free (FMF) systems.

The final decision is a weighted integration of these two systems, governed by a parameter $\omega$. A high $\omega$ makes the agent more Pavlovian (a "sign-tracker"), while a low $\omega$ makes it more goal-directed.

Task Representation and Methods

The authors employ a dual-learning framework using Markov Decision Processes (MDPs) to simulate the experimental environment. By associating value with features (the light itself) rather than just states (the room with the light on), the model explains why pigeons are "drawn" to the light.

State transition diagram of the negative automaintenance task modeled as an MDP.

As shown in the state transition diagram, the agent moves through states representing the presentation of the key light ($s_1$), the choice to peck or refrain ($s_2, s_3, s_4$), and the eventual outcome ($s_0$). The model was validated against classic behavioral data from Williams & Williams (1969) and Sanabria et al. (2006) to qualitatively replicate the cumulative response curves and behavioral shifts observed in pigeons.

Key Results and Insights

  • The $\omega$ Parameter as a Biological Trait: The model suggests that the difference between a "smart" pigeon and a "stubborn" one is simply the weighting between these two neural systems. With a high $\omega$ (0.9), simulated pigeons pecked the light in ~50% of trials, matching the "sign-tracking" behavior. With a low $\omega$ (0.2), they quickly learned to refrain, matching "goal-tracking" behavior.
  • Avoidance Strategies: The model successfully simulated experiments where pigeons shifted their pecks to a second "irrelevant" light that also predicted food but didn't cancel the reward. The FMF system was "distracted" by the new predictive feature, allowing the MB system to maximize reward.
  • The "Continuous Key" Effect: The model explains why a light that is always on is less attractive than one that flashes: stimuli present during the inter-trial interval (ITI) lose their predictive value in the FMF system, reducing the "urge" to peck them.

Limitations and Future Work

While the model provides a robust qualitative framework, it uses a discrete-time MDP, meaning it cannot account for the exact timing or latency of pecks. Furthermore, parameters were tuned manually to match qualitative data. Future work could use automated fitting if more granular raw data becomes available. The authors also provide specific predictions for Dopamine (DA) activity—expecting DA peaks at the light's appearance and "dips" when rewards are omitted—which could be verified using in-vivo recordings.

Summary updated

Abstract

Animals, including Humans, are prone to develop persistent maladaptive and suboptimal behaviours. Some of these behaviours have been suggested to arise from interactions between brain systems of Pavlovian conditioning, the acquisition of responses to initially neutral stimuli previously paired with rewards, and instrumental conditioning, the acquisition of active behaviours leading to rewards. However the mechanics of these systems and their interactions are still unclear. While extensively studied independently, few models have been developed to account for these interactions. On some experiment, pigeons have been observed to display a maladaptive behaviour that some suggest to involve conflicts between Pavlovian and instrumental conditioning. In a procedure referred as negative automaintenance, a key light is paired with the subsequent delivery of food, however any peck towards the key light results in the omission of the reward. Studies showed that in such procedure some pigeons persisted in pecking to a substantial level despite its negative consequence, while others learned to refrain from pecking and maximized their cumulative rewards. Furthermore, the pigeons that were unable to refrain from pecking could nevertheless shift their pecks towards a harmless alternative key light. We confronted a computational model that combines dual-learning systems and factored representations, recently developed to account for sign-tracking and goal-tracking behaviours in rats, to these negative automaintenance experimental data. We show that it can explain the variability of the observed behaviours and the capacity of alternative key lights to distract pigeons from their detrimental behaviours. These results confirm the proposed model as an interesting tool to reproduce experiments that could involve interactions between Pavlovian and instrumental conditioning. The model allows us to draw predictions that may be experimentally verified, which could help further investigate the neural mechanisms underlying theses interactions.
BibTeX
@inproceedings{lesaint2014accounting,
  author = {Florian Lesaint and Olivier Sigaud and Mehdi Khamassi},
  title = {{Accounting for Negative Automaintenance in Pigeons: A Dual Learning Systems Approach and Factored Representations}},
  booktitle = {PLoS},
  year = {2014},
  doi = {10.1371/journal.pone.0111050},
}
Figure