Random Exploration is Random

Yingkang Luo, Hua-Dong Xiong, Hanbo Xie, Caroline Phelps, Madeline Harms, Ryan Smith, James Waltz, Robert WilsonarXiv 2026 · arXiv

TL;DR

When faced with the "explore-exploit" dilemma, humans become more variable in their choices as the value of information increases. This study uses deep neural networks to show that this increased variability isn't just a failure of simple models to capture complex patterns—it appears to be fundamentally stochastic, suggesting that "random exploration" is truly random.

What Problem Does This Paper Address?

In decision-making, we constantly balance exploitation (choosing the best known option) with exploration (trying something new to learn more). Researchers have identified two main exploratory strategies: directed exploration, where we deliberately seek out information, and random exploration, where our choices simply become noisier and less predictable.

The big question is: Is that "noise" actually random?

Traditional cognitive models use a "decision noise" parameter to mop up everything they can't explain. Skeptics argue that this noise might just be "unmodeled structure"—complex, deterministic strategies involving long-term history or non-linear patterns that simple models are too "dumb" to see. If we use more powerful models, like deep neural networks, can we "explain away" this randomness?

Methods

The authors used the Horizon Task, a classic paradigm where participants choose between two slot machines. The key variable is the "horizon": how many future choices they have left. In a short horizon (H1), information is useless because the game ends immediately. In a long horizon (H6), information is valuable for future turns.

The Horizon Task setup, showing instructed trials followed by free-choice trials in H1 and H6 conditions.

To test the limits of predictability, the researchers compared a baseline logistic regression model against a suite of high-capacity neural networks (NNs):

  • Multilayer Perceptrons (MLP): To capture non-linear relationships in the current game.
  • Recurrent Neural Networks (RNN, LSTM, GRU): To capture sequential dependencies.
  • Transformers: To model flexible dependencies across trials.
  • Cross-game RNNs: To see if behavior is influenced by the history of previous games.

They evaluated these models at both a "global" (pooled population) level and a "subject-level" (using one-hot encoding for individual identities).

Datasets

The study utilized a massive aggregated dataset:

  • 1,392 participants across multiple prior studies.
  • Over 1.27 million trials (169,376 total games).
  • Diverse demographics, with ages ranging from 10 to 74 years.

Key Insights and Ideas

  • The Persistence of the Gap: While neural networks were significantly better at predicting behavior than the baseline model (increasing McFadden’s pseudo-$R^2$ from 0.217 to ~0.385), they could not close the "predictability gap." Choices in the long-horizon (H6) condition remained consistently harder to predict than those in the short-horizon (H1) condition.
  • Strategic Randomness: The fact that NNs—which are excellent at finding hidden patterns—couldn't eliminate the extra noise in H6 suggests that humans strategically increase their internal stochasticity when exploration is valuable.
  • History Doesn't Explain It: Even models that tracked hidden states across multiple games couldn't account for the H6 variability. This implies the "randomness" isn't a slow-moving strategy or a learning effect, but a trial-by-trial injection of noise.
  • Predictability Increases as the End Nears: In H6 games, choices become more predictable as the remaining horizon shortens. This confirms that the "randomness" is tightly coupled to the potential value of future exploration.
Predictive accuracy increases as the remaining horizon in H6 games shortens, though it never quite reaches the H1 benchmark.

Key Results

  • Baseline Noise: The traditional model showed a significant increase in the "decision noise" parameter ($\eta_d$) for H6 compared to H1.
Baseline model parameter estimates showing increased information bonus and decision noise in long-horizon conditions.
  • Accuracy Gap: Across all NN architectures, H1 test accuracy hovered around 81-82%, while H6 accuracy was significantly lower, around 73-76%.
  • Subject-Level Gains: Including subject identity improved overall accuracy but the ~6-8% gap between H1 and H6 remained stubbornly present across all models, including Transformers and LSTMs.
Comparison of test accuracy across different NN architectures for H1 (blue) and H6 (red), showing the persistent gap.

Limitations and Future Work

The authors acknowledge that "random" might just mean "unmeasured." While the NNs had access to choices and rewards, they didn't have access to:

  • Physiological data: Gaze patterns (eye tracking), pupil dilation (arousal), or fidgeting/micro-movements.
  • Internal states: Boredom, attention lapses, or specific cognitive "heuristics" not captured by the task variables.

Future work could integrate these richer data streams to see if the "residual randomness" can be further reduced. Additionally, the authors suggest moving toward "interpretable" machine learning to understand exactly what behavioral structures the NNs are picking up that the logistic models miss.

Summary updated

Abstract

Decision making often requires balancing exploitation of familiar options with exploration of uncertain alternatives. One strategy for exploration is random exploration, in which choices become more variable and less tied to the option that currently appears best. However, it remains unclear whether this variability reflects genuine stochasticity or instead a deterministic structure that traditional cognitive models fail to capture. We examined this question using the Horizon Task in an aggregated dataset of 1,392 participants completing over 1,270,695 trials. In this task, exploration becomes more valuable when participants have more future choices, and choice variability increases under longer horizons. We tested whether more expressive neural network models could recover deterministic structure from this additional variability. Although neural networks provided better overall fits to behavior than the traditional model, long-horizon choices remained consistently less predictable than short-horizon choices. These findings show that limited model expressiveness cannot fully account for the increased variability associated with random exploration, suggesting random exploration truly is substantially random.
BibTeX
@misc{luo2026random,
  author = {Yingkang Luo and Hua-Dong Xiong and Hanbo Xie and Caroline Phelps and Madeline Harms and Ryan Smith and James Waltz and Robert Wilson},
  title = {{Random Exploration is Random}},
  year = {2026},
  eprint = {2611.17011},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17011},
}
Figure