The role of training variability for model-based and model-free learning of an arbitrary visuomotor mapping

Carlos A. Velázquez-Vargas, Nathaniel D. Daw, Jordan A. Taylor

TL;DR

This paper investigates how training variability impacts the learning of arbitrary visuomotor mappings, crucial for skills like video games or robotics. They found that varied training leads to more flexible, generalizable learning, likely by promoting model-based reasoning over habitual responses. This has implications for designing more effective training protocols for complex motor skills.

What Problem Does This Paper Address?

Learning new motor skills often involves mastering arbitrary mappings between actions and outcomes – think of controlling a character in a video game with a joystick or a robotic arm with a new interface. While we know that practice leads to skill acquisition, the way we practice significantly influences how well we learn and how flexibly we can apply that learning to new situations. Specifically, this paper asks: how does the variability in training (e.g., practicing with many different start-target pairs versus just one) affect our ability to learn and generalize an arbitrary visuomotor mapping? They hypothesize that higher training variability encourages a more robust, generalizable "mental model" of the mapping, rather than just rote memorization of specific action sequences.

Methods

The researchers used a grid navigation task where participants controlled a cursor (a ship) to move from a start to a target location using keyboard keys. The core of the study involved manipulating training variability:

  • Single Group: Trained on a single start-target pair.
  • Multiple Group: Trained on multiple start-target pairs.

They employed a combination of behavioral experiments and computational modeling:

  • Behavioral Experiments: Measured performance through optimal arrivals (reaching the target in the minimum number of moves) and reaction times (RTs). They tested different generalization scenarios (targets further or closer, with or without feedback) and introduced stochasticity (randomness) in the mapping to explore explicit vs. implicit learning.
  • Computational Modeling: Evaluated five reinforcement learning models:
  • Model-Free (MF): Learns based on prediction errors, focusing on immediate rewards (e.g., reducing distance to the target).
  • Model-Based (MB): Learns the underlying mapping and uses it for planning to find optimal paths.
  • Hybrid Models (1W, 2W, 4W): Combinations of MF and MB with different weighting schemes for training and generalization phases.
  • Arbitration Model (AR): A hybrid model where the weight of the model-based component dynamically adjusts based on the familiarity of encountered states.

Datasets

The study did not use pre-existing datasets. Instead, they collected data from human participants performing a custom-designed grid navigation task. This task, based on prior work, was adapted to specifically investigate visuomotor mapping learning under varying training conditions.

Key Insights and Ideas

  • Variability is Key for Generalization: Training with multiple start-target pairs (high variability) significantly improved participants' ability to generalize their learned mapping to novel targets, especially when planning was required. The "Single" group, trained on only one pair, struggled to adapt.
  • Model-Based Reasoning is Promoted by Variability: During training, participants in the "Multiple" group were more likely to make "counterintuitive" moves that minimized the model-based distance (i.e., reduced the number of key presses needed) even if it meant moving visually away from the target. This suggests they were actively using their learned mapping for planning, not just relying on simple stimulus-response associations.
  • Habits Can Hinder Flexibility: The "Single" group's poor generalization performance, even in simple scenarios, suggests that overtraining a single mapping can lead to habitual responses that interfere with flexible use of the learned skill.
  • Early Variability is Powerful: A short burst of varied training early in learning, or even introducing stochasticity into the mapping, could enhance generalization for the "Single" group, suggesting that preventing rote memorization early on is crucial.
  • Hybrid Models Best Explain Behavior: Computational modeling revealed that a hybrid model, allowing for different contributions of model-based and model-free systems during training and generalization, best captured human behavior. The "Multiple" group consistently showed higher model-based weights, especially during generalization.

Key Results

  • Experiment 1 & 2: The "Multiple" group showed significantly better generalization performance to novel targets compared to the "Single" group, particularly when planning was involved. While both groups learned the mapping during training, the "Single" group's performance declined in generalization.
  • Experiment 3: A short period of varied training followed by extensive single-pair training still allowed for good generalization, suggesting that early formation of a robust mapping can be retained.
  • Experiment 4: Introducing stochasticity into the mapping (making it less predictable) improved generalization for the "Single" group to levels comparable to the "Multiple" group, suggesting it disrupted habitual response formation.
  • Modeling: The 2W (two-weight) hybrid model consistently outperformed others in explaining individual and group data, with the "Multiple" group showing higher model-based weights during generalization.

Limitations and Future Work

  • Habit Formation: The authors acknowledge that their current models don't fully capture the persistence of habitual responses, which could be a factor in the "Single" group's performance. Future work could incorporate persistence mechanisms.
  • Planning Depth: The model-based algorithm assumes infinite planning depth, which is unrealistic. Future models could incorporate planning depth limitations.
  • Explicit vs. Implicit Learning: While Experiment 4 explored stochasticity to blunt explicit learning, further research is needed to fully disentangle the roles of explicit and implicit processes in visuomotor mapping acquisition.
  • Task Complexity: The current mapping was relatively simple. Future studies could explore more complex mappings (e.g., a "Knight's move" in chess) to better understand the acquisition of complex skills.

Summary updated

Abstract

A fundamental feature of the human brain is its capacity to learn novel motor skills. This capacity requires the formation of vastly different visuomotor mappings. Using a grid navigation task, we investigated whether training variability would enhance the flexible use of a visuomotor mapping (key-to-direction rule), leading to better generalization performance. Experiments 1 and 2 show that participants trained to move between multiple start-target pairs exhibited greater generalization to both distal and proximal targets compared to participants trained to move between a single pair. This finding suggests that limited variability can impair decisions even in simple tasks without planning. In addition, during the training phase, participants exposed to higher variability were more inclined to choose options that, counterintuitively, moved the cursor away from the target while minimizing its actual distance under the constrained mapping, suggesting a greater engagement in model-based computations. In Experiments 3 and 4, we showed that the limited generalization performance in participants trained with a single pair can be enhanced by a short period of variability introduced early in learning or by incorporating stochasticity into the visuomotor mapping. Our computational modeling analyses revealed that a hybrid model between model-free and model-based computations with different mixing weights for the training and generalization phases, best described participants’ data. Importantly, the differences in the model-based weights between our experimental groups, paralleled the behavioral findings during training and generalization. Taken together, our results suggest that training variability enables the flexible use of the visuomotor mapping, potentially by preventing the consolidation of habits due to the continuous demand to change responses.
BibTeX
@article{velazquezvargas2024role,
  author = {Carlos A. Velázquez-Vargas and Nathaniel D. Daw and Jordan A. Taylor},
  title = {{The role of training variability for model-based and model-free learning of an arbitrary visuomotor mapping}},
  journal = {PLOS Computational Biology},
  year = {2024},
  doi = {10.1371/journal.pcbi.1012471},
}
Figure