Identifying category representations for complex stimuli using discrete Markov chain Monte Carlo with people

Anne Hsu, Jay B. Martin, Adam N. Sanborn, Thomas L. GriffithsBehavior 2019 · DOI

Abstract

With the explosion of "big data," digital repositories of texts and images are growing rapidly. These datasets present new opportunities for psychological research, but they require new methodologies before researchers can use these datasets to yield insights into human cognition. We present a new method that allows psychological researchers to take advantage of text and image databases: a procedure for measuring human categorical representations over large datasets of items, such as arbitrary words or pictures. We call this method discrete Markov chain Monte Carlo with people (d-MCMCP). We illustrate our method by evaluating the following categories over datasets: emotions as represented by facial images, moral concepts as represented by relevant words, and seasons as represented by images drawn from large online databases. Three experiments demonstrate that d-MCMCP is powerful and flexible enough to work with complex, naturalistic stimuli drawn from large online databases.

Introduction

Understanding how humans represent mental categories—such as what constitutes a "typical" face or a "moral" concept—is a cornerstone of psychological research. Traditionally, researchers have relied on participants providing absolute ratings for stimuli on fixed scales. However, this approach faces significant scalability issues in the era of "Big Data." Rating thousands of images or words is prohibitively time-consuming for human participants. Furthermore, cognitive science suggests that humans are far more accurate at making relative, pairwise comparisons than providing absolute numerical judgments.

While previous iterations of Markov chain Monte Carlo with people (MCMCP) successfully navigated continuous, parameterized stimulus spaces (e.g., morphed faces), they were unable to handle discrete, real-world datasets like collections of natural images or arbitrary word lists. This paper introduces discrete Markov chain Monte Carlo with people (d-MCMCP), a framework designed to extract psychological insights from large-scale, unparameterized discrete datasets.

Methodology: The d-MCMCP Framework

The d-MCMCP method adapts the Metropolis-Hastings algorithm—a standard tool in computational statistics—to incorporate human decision-making as the "acceptance function." The process is structured into four distinct phases:

  1. Database Assembly: A large corpus of discrete items, such as images or text, is gathered.
  2. Similarity Mapping: An automated metric (e.g., Gabor filters for images or Latent Semantic Analysis for text) is used to compute a rough similarity matrix across the dataset.
  3. Graph Construction ($b$-matching): To ensure the algorithm can traverse the dataset effectively, items are organized into a graph where each item is connected to its $b$ most similar neighbors. A "maximum-weight $b$-matching" algorithm is employed to ensure the graph is symmetric and regular, a mathematical prerequisite for the Markov chain to converge correctly.
  4. Human Acceptance Step: Participants are presented with a "current" item and a "proposal" item (a neighbor in the graph) and asked to choose which better represents a target category.
Overview of the d-MCMCP methodology: from database assembly to the human-in-the-loop experiment.

Mathematically, if human choices follow the Luce choice rule, the resulting Markov chain converges to a stationary distribution that reflects the participant's internal mental representation of the category.

Experimental Validation

The authors validated d-MCMCP across three diverse domains:

Experiment 1: Facial Expressions

Using 1,271 portraits from the California Facial Expression (CAFE) database, the method produced "average" happy and sad faces. These representations were found to be more representative of the target emotions than those generated by traditional continuous-space methods like eigenfaces.

Experiment 2: Moral Concepts

The method was applied to a dictionary of 160 words related to Moral Foundations Theory. The results successfully mapped ideological differences: conservative participants' chains sampled words related to "purity" significantly more often than those of liberal participants, who focused primarily on "harm" and "fairness."

Experiment 3: Seasonal Imagery

In a noisy dataset of 4,000 images scraped from the web, d-MCMCP converged on distinct color palettes and iconic imagery (e.g., snow for winter, blossoms for spring) that intuitively aligned with seasonal categories.

Discussion

A key strength of d-MCMCP is its efficiency through sparsity; the algorithm naturally gravitates toward representative items, ignoring irrelevant noise without requiring exhaustive ratings. Furthermore, the method is robust to initial metrics; the automated similarity mapping only needs to be "good enough" to guide the chain, as human judgments provide the ultimate ground truth.

However, limitations remain. The results are inherently tied to the biases of the initial dataset. Additionally, while more efficient than exhaustive methods, the chains still require a significant number of trials to reach convergence. Future research may explore optimizing graph structures or applying the method to even more complex stimuli like video or consumer preferences.

Summary updated

Abstract

With the explosion of "big data," digital repositories of texts and images are growing rapidly. These datasets present new opportunities for psychological research, but they require new methodologies before researchers can use these datasets to yield insights into human cognition. We present a new method that allows psychological researchers to take advantage of text and image databases: a procedure for measuring human categorical representations over large datasets of items, such as arbitrary words or pictures. We call this method discrete Markov chain Monte Carlo with people (d-MCMCP). We illustrate our method by evaluating the following categories over datasets: emotions as represented by facial images, moral concepts as represented by relevant words, and seasons as represented by images drawn from large online databases. Three experiments demonstrate that d-MCMCP is powerful and flexible enough to work with complex, naturalistic stimuli drawn from large online databases.
BibTeX
@inproceedings{hsu2019identifying,
  author = {Anne Hsu and Jay B. Martin and Adam N. Sanborn and Thomas L. Griffiths},
  title = {{Identifying category representations for complex stimuli using discrete Markov chain Monte Carlo with people}},
  booktitle = {Behavior},
  year = {2019},
  doi = {10.3758/s13428-019-01201-9},
}
Figure