Ten simple rules for the computational modeling of behavioral data

Robert C. Wilson, Anne CollinseLife 2019 · DOI

TL;DR

Computational modeling is a powerful tool for understanding the mind, but it's easy to misuse. This paper provides ten practical, beginner-friendly rules to ensure models are used effectively, leading to meaningful insights rather than misleading conclusions. By following these guidelines, researchers can avoid common pitfalls and unlock the true potential of computational modeling in behavioral science.

What Problem Does This Paper Address?

The field of computational modeling in psychology and neuroscience has exploded, offering unprecedented ways to probe the mechanisms underlying behavior. However, this power comes with a significant responsibility. Researchers can easily fall into traps, leading to incorrect interpretations or over-reliance on models that don't truly capture the data. The challenge is to provide a clear, actionable framework for using computational models correctly, especially for those new to the field, ensuring that modeling efforts yield genuine scientific progress.

Methods

The paper outlines ten simple rules, presented as a process in Figure 1, for computational modeling of behavioral data. These rules cover the entire workflow, from experimental design to reporting results. The authors emphasize a pragmatic, details-oriented approach, illustrating their points with simple reinforcement learning models applied to a multi-armed bandit task. Key techniques discussed include:

  • Experimental Design: Ensuring the experiment elicits behaviors that can be captured by the model and that the targeted processes are evident in basic statistics.
  • Model Building: Designing models that are as simple as possible but no simpler, interpretable, and capable of capturing competing hypotheses.
  • Simulation: Generating "fake" data to understand model behavior across parameter ranges and to visualize qualitative differences between models.
  • Parameter Estimation: Primarily focusing on maximum likelihood estimation (MLE) to find model parameters that best fit the data.
  • Model Comparison: Using metrics like the Bayesian Information Criterion (BIC) to select the best-fitting model among a set of candidates.
  • Parameter Recovery: A crucial step to ensure that the fitting procedure can accurately recover known parameter values from simulated data.
  • Model Validation: Checking if the winning model can reproduce qualitative and quantitative patterns in the data, not just fit the data it was trained on.
  • Latent Variable Analysis: Extracting hidden variables from the model to understand internal cognitive processes and relate them to neural data.
  • Reporting: Emphasizing transparency by sharing data and code, and reporting results comprehensively.

The paper provides code examples and detailed explanations for each step, making it a practical guide.

Datasets

Not discussed in the paper. The paper uses simulated data for illustrative purposes and refers to published findings and publicly available datasets as sources for specialist modelers.

Key Insights and Ideas

  • Good experimental design is paramount: No amount of sophisticated modeling can salvage data from a poorly designed experiment. The experiment must be capable of revealing the phenomena you aim to model.
  • Simulate, simulate, simulate! Before fitting models to real data, thoroughly explore how your models behave with simulated data. This builds intuition, helps refine models and experiments, and is crucial for understanding parameter interpretability.
  • Parameter recovery is non-negotiable: If your model can't recover known parameters from simulated data, it's unlikely to provide meaningful insights from real data. This step acts as a sanity check for your entire modeling pipeline.
  • Model validation is more than just goodness-of-fit: A model that fits data well quantitatively might still fail to capture the qualitative essence of the behavior. Validation ensures the model truly reflects the underlying processes.
  • Transparency is key: Sharing data and code is essential for reproducibility, collaboration, and advancing the field.

Key Results

The paper doesn't present empirical results from a specific experiment but rather provides a methodological framework. The "results" are the insights gained from applying these rules to illustrative examples, such as:

  • Demonstrating how different models (e.g., random responding, win-stay-lose-shift, Rescorla-Wagner) make distinct predictions in a bandit task.
  • Illustrating the importance of parameter exploration in simulation to understand how parameters influence behavior.
  • Showing how parameter recovery can reveal issues with model identifiability or experimental design.
  • Highlighting how including "unimportant" parameters (like a side bias) can significantly improve the recovery of important parameters.
  • Providing examples of model validation where a model that fits well quantitatively fails to capture essential behavioral patterns.

Limitations and Future Work

  • Model Complexity: While the paper focuses on simpler models for accessibility, it acknowledges that more advanced techniques exist and have their own complexities and pitfalls.
  • Model Selection: The paper emphasizes that model comparison only tells you which considered model is best, not whether it's the "true" model. This underscores the importance of starting with a good set of competing hypotheses.
  • Future Work: The authors point towards advanced techniques like Bayesian model selection, incorporating other data types (e.g., reaction times, eye movements, neural data), and using more sophisticated optimizers and approximate likelihood methods. They also suggest improving models to account for discrepancies and using models to make predictions for new experiments.

Summary updated

Abstract

Computational modeling of behavior has revolutionized psychology and neuroscience. By fitting models to experimental data we can probe the algorithms underlying behavior, find neural correlates of computational variables and better understand the effects of drugs, illness and interventions. But with great power comes great responsibility. Here, we offer ten simple rules to ensure that computational modeling is used with care and yields meaningful insights. In particular, we present a beginner-friendly, pragmatic and details-oriented introduction on how to relate models to data. What, exactly, can a model tell us about the mind? To answer this, we apply our rules to the simplest modeling techniques most accessible to beginning modelers and illustrate them with examples and code available online. However, most rules apply to more advanced techniques. Our hope is that by following our guidelines, researchers will avoid many pitfalls and unleash the power of computational modeling on their own data.
BibTeX
@inproceedings{wilson2019ten,
  author = {Robert C. Wilson and Anne Collins},
  title = {{Ten simple rules for the computational modeling of behavioral data}},
  booktitle = {eLife},
  year = {2019},
  doi = {10.7554/elife.49547},
}
Figure