TL;DR
When faced with the "explore-exploit" dilemma, humans become more variable in their choices as the value of information increases. This study uses deep neural networks to show that this increased variability isn't just a failure of simple models to capture complex patterns—it appears to be fundamentally stochastic, suggesting that "random exploration" is truly random.
What Problem Does This Paper Address?
In decision-making, we constantly balance exploitation (choosing the best known option) with exploration (trying something new to learn more). Researchers have identified two main exploratory strategies: directed exploration, where we deliberately seek out information, and random exploration, where our choices simply become noisier and less predictable.
The big question is: Is that "noise" actually random?
Traditional cognitive models use a "decision noise" parameter to mop up everything they can't explain. Skeptics argue that this noise might just be "unmodeled structure"—complex, deterministic strategies involving long-term history or non-linear patterns that simple models are too "dumb" to see. If we use more powerful models, like deep neural networks, can we "explain away" this randomness?
Methods
The authors used the Horizon Task, a classic paradigm where participants choose between two slot machines. The key variable is the "horizon": how many future choices they have left. In a short horizon (H1), information is useless because the game ends immediately. In a long horizon (H6), information is valuable for future turns.
To test the limits of predictability, the researchers compared a baseline logistic regression model against a suite of high-capacity neural networks (NNs):
- Multilayer Perceptrons (MLP): To capture non-linear relationships in the current game.
- Recurrent Neural Networks (RNN, LSTM, GRU): To capture sequential dependencies.
- Transformers: To model flexible dependencies across trials.
- Cross-game RNNs: To see if behavior is influenced by the history of previous games.
They evaluated these models at both a "global" (pooled population) level and a "subject-level" (using one-hot encoding for individual identities).
Datasets
The study utilized a massive aggregated dataset:
- 1,392 participants across multiple prior studies.
- Over 1.27 million trials (169,376 total games).
- Diverse demographics, with ages ranging from 10 to 74 years.
Key Insights and Ideas
- The Persistence of the Gap: While neural networks were significantly better at predicting behavior than the baseline model (increasing McFadden’s pseudo-$R^2$ from 0.217 to ~0.385), they could not close the "predictability gap." Choices in the long-horizon (H6) condition remained consistently harder to predict than those in the short-horizon (H1) condition.
- Strategic Randomness: The fact that NNs—which are excellent at finding hidden patterns—couldn't eliminate the extra noise in H6 suggests that humans strategically increase their internal stochasticity when exploration is valuable.
- History Doesn't Explain It: Even models that tracked hidden states across multiple games couldn't account for the H6 variability. This implies the "randomness" isn't a slow-moving strategy or a learning effect, but a trial-by-trial injection of noise.
- Predictability Increases as the End Nears: In H6 games, choices become more predictable as the remaining horizon shortens. This confirms that the "randomness" is tightly coupled to the potential value of future exploration.
Key Results
- Baseline Noise: The traditional model showed a significant increase in the "decision noise" parameter ($\eta_d$) for H6 compared to H1.
- Accuracy Gap: Across all NN architectures, H1 test accuracy hovered around 81-82%, while H6 accuracy was significantly lower, around 73-76%.
- Subject-Level Gains: Including subject identity improved overall accuracy but the ~6-8% gap between H1 and H6 remained stubbornly present across all models, including Transformers and LSTMs.
Limitations and Future Work
The authors acknowledge that "random" might just mean "unmeasured." While the NNs had access to choices and rewards, they didn't have access to:
- Physiological data: Gaze patterns (eye tracking), pupil dilation (arousal), or fidgeting/micro-movements.
- Internal states: Boredom, attention lapses, or specific cognitive "heuristics" not captured by the task variables.
Future work could integrate these richer data streams to see if the "residual randomness" can be further reduced. Additionally, the authors suggest moving toward "interpretable" machine learning to understand exactly what behavioral structures the NNs are picking up that the logistic models miss.
Summary updated