A tutorial on open-source large language models for behavioral science

Z. Hussain, Marcel Binz, Rui Mata, Dirk U. Wulff

Abstract

Large language models (LLMs) have the potential to revolutionize behavioral science by accelerating and improving the research cycle, from conceptualization to data analysis. Unlike closed-source solutions, open-source frameworks for LLMs can enable transparency, reproducibility, and adherence to data protection standards, which gives them a crucial advantage for use in behavioral science. To help researchers harness the promise of LLMs, this tutorial offers a primer on the open-source Hugging Face ecosystem and demonstrates several applications that advance conceptual and empirical work in behavioral science, including feature extraction, fine-tuning of models for prediction, and generation of behavioral responses. Executable code is made available at github.com/Zak-Hussain/LLM4BeSci.git. Finally, the tutorial discusses challenges faced by research with (open-source) LLMs related to interpretability and safety and offers a perspective on future research at the intersection of language modeling and behavioral science.

Introduction

Behavioral science stands at a transformative juncture with the advent of Large Language Models (LLMs). While these models offer unprecedented capabilities for analyzing linguistic data and simulating human behavior, the field's reliance on proprietary, "black-box" APIs presents significant challenges. Hussain et al. provide a practical roadmap for researchers to transition toward transparent, reproducible, and privacy-preserving open-source workflows using the Hugging Face ecosystem.

The Open-Source Advantage

The shift from closed-source models (e.g., OpenAI’s GPT series) to open-source alternatives addresses three critical hurdles in behavioral research:

  1. Transparency and Reproducibility: Proprietary models are "moving targets" with unknown weights and training data, making scientific replication nearly impossible.
  2. Data Privacy: Ethical and legal constraints often prohibit uploading sensitive clinical or personal data to third-party corporate servers.
  3. Technical Accessibility: Open-source tools bridge the gap between traditional psychological methods and complex machine learning engineering.

Methodological Framework

The tutorial introduces the Hugging Face ecosystem (specifically the transformers, datasets, and accelerate libraries) as the standard toolkit for open-source LLM research. It categorizes LLM applications into four distinct methodological pipelines:

High-level pipeline of an LLM workflow, illustrating the path from input prompt to output probabilities.
  • Feature Extraction: Utilizing "frozen" models to convert text into high-dimensional numerical vectors (embeddings). These embeddings serve as predictors in traditional statistical frameworks, such as Ridge regression.
  • Fine-Tuning: Updating pre-trained model weights on specialized behavioral datasets to optimize performance for specific tasks.
  • Text Generation: Prompting models to produce human-like text for experimental stimuli or "in-silico" participant simulation.
  • Log-Probability Extraction: Measuring the "surprisingness" or perplexity of text based on the model's internal probability distributions.

Architecture and Implementation

A key insight for researchers is the distinction between model architectures. Encoders (e.g., BERT) are generally superior for embedding and classification tasks, while Decoders (e.g., Llama, GPT) excel at generative tasks.

Detailed internal architecture of a transformer block, showing the multi-head attention mechanism and feed-forward layers.

Furthermore, the paper highlights the utility of "quantization," which allows large models (e.g., Llama 2 13B) to run on standard consumer-grade GPUs. This enables local processing of sensitive data, ensuring it never leaves the researcher's laboratory.

Empirical Validations

The authors validate these methods across several behavioral domains:

  • Personality Mapping: Using DistilBERT, the authors demonstrated that embeddings can capture the semantic structure of the "Big Five" personality factors from the IPIP-NEO-300 inventory without manual coding.
  • Health Perception: Combining DistilBERT embeddings with Ridge regression explained 58% of the variance ($R^2 = .58$) in human health severity ratings. Larger models like instructor-xl improved this to $R^2 = .70$.
  • Decision Making: In the Horizon Task, a quantized Llama 2 model predicted human choices with 82.9% accuracy, matching the performance of specialized psychological models.
  • Linguistic Readability: Perplexity measures from GPT-2 accounted for 20% of the variance in human-assigned readability scores in the CLEAR corpus.

Challenges and Future Directions

Despite their promise, LLMs present ongoing challenges:

  • Interpretability: The "black-box" nature of high-dimensional embeddings remains a hurdle for theoretical understanding.
  • Data Contamination: "Data leakage" occurs when experimental tasks (like the Cognitive Reflection Test) are present in the model's pre-training data, potentially inflating performance.
  • Compute Requirements: While quantization lowers the barrier, state-of-the-art models still require substantial hardware resources.

Conclusion

By adopting open-source LLMs, behavioral scientists can enhance the transparency and privacy of their research while leveraging powerful new tools for feature extraction and behavioral simulation. The provided code repository serves as a foundational resource for researchers entering this space.

Code Availability: github.com/Zak-Hussain/LLM4BeSci

Summary updated

Abstract

Large language models (LLMs) have the potential to revolutionize behavioral science by accelerating and improving the research cycle, from conceptualization to data analysis. Unlike closed-source solutions, open-source frameworks for LLMs can enable transparency, reproducibility, and adherence to data protection standards, which gives them a crucial advantage for use in behavioral science. To help researchers harness the promise of LLMs, this tutorial offers a primer on the open-source Hugging Face ecosystem and demonstrates several applications that advance conceptual and empirical work in behavioral science, including feature extraction, fine-tuning of models for prediction, and generation of behavioral responses. Executable code is made available at github.com/Zak-Hussain/LLM4BeSci.git. Finally, the tutorial discusses challenges faced by research with (open-source) LLMs related to interpretability and safety and offers a perspective on future research at the intersection of language modeling and behavioral science.
BibTeX
@inproceedings{hussain2024tutorial,
  author = {Z. Hussain and Marcel Binz and Rui Mata and Dirk U. Wulff},
  title = {{A tutorial on open-source large language models for behavioral science}},
  booktitle = {Behavior},
  year = {2024},
  doi = {10.3758/s13428-024-02455-8},
}
Figure