Abstract
Humans sometimes show sudden improvements in task performance that have been linked to moments of insight. Such insight-related performance improvements appear special because they are preceded by an extended period of impasse, are unusually abrupt, and occur only in some, but not all, learners. Here, we ask whether insight-like behaviour also occurs in artificial neural networks trained with gradient descent algorithms. We compared learning dynamics in humans and regularised neural networks in a perceptual decision task that provided a hidden opportunity which allowed to solve the task more efficiently. We show that humans tend to discover this regularity through insight, rather than gradually. Notably, neural networks with regularised gate modulation closely mimicked behavioural characteristics of human insights, exhibiting delay of insight, suddenness and selective occurrence. Analyses of network learning dynamics revealed that insight-like behaviour crucially depended on noise added to gradient updates, and was preceded by "silent knowledge" that is initially suppressed by regularised (attentional) gating. This suggests that insights can arise naturally from gradual learning, where they reflect the combined influences of noise, attentional gating and regularisation.
Introduction
In psychology, "insight" is often characterized as a mysterious, qualitatively distinct form of learning—the classic "Aha!" moment. Unlike the gradual refinement of skills typical of most artificial intelligence models, human insight is marked by a sudden leap in performance following a period of "impasse," where the solution seems entirely out of reach.
Traditionally, these leaps have been attributed to high-level cognitive processes such as representational restructuring, meta-cognitive monitoring, or the transition from unconscious to conscious processing. However, recent research challenges this view, asking whether the behavioral hallmarks of insight—suddenness, selectivity, and variable delay—can emerge from simple, gradual learning rules within a regularized system.
The Spontaneous Strategy Switch Task
To investigate this, researchers compared human behavior with a minimal neural network architecture using a "Spontaneous Strategy Switch Task."
In this task, participants (and networks) are asked to categorize moving colored dots. Initially, only the motion of the dots is predictive of the correct category, though this signal is noisy and difficult to discern. Unbeknownst to the subjects, a hidden regularity is eventually introduced: the color of the dots becomes a perfect, easy-to-use predictor. The core question is how, and if, learners switch from the difficult motion-based strategy to the efficient color-based strategy.
Model Architecture: The "Secret Sauce"
The study utilized a "stripped-down" neural network with two input nodes (motion and color) and one output node. To mimic human-like insight, the researchers implemented three specific features:
- Multiplicative Gating: Each input is governed by a "gate" that modulates its weight, functioning similarly to biological attentional filtering.
- L1-Regularization: A penalty is applied to the gates based on their absolute value. This encourages sparsity, effectively keeping "irrelevant" gates closed until a strong enough signal is present.
- Noisy SGD: Gaussian noise is added to the gradient updates to induce trial-to-trial variability, reflecting the stochastic nature of biological learning.
Results and Behavioral Matching
The L1-regularized networks demonstrated a remarkable behavioral match to human participants. Approximately 50.5% of humans and 46.5% of L1-networks were classified as having achieved "insight."
Key behavioral metrics included:
- Suddenness: In both insightful humans and networks, the transition from the old strategy to the new one was nearly instantaneous, often occurring within a few trials (roughly 15–30 seconds for humans).
- Delay: Both groups experienced a variable "impasse" period. Networks showed an average delay of 175 trials, while humans averaged 130 trials.
- Selectivity: Not all subjects reached the insight. The researchers found that random noise in gradient updates allowed some models to accumulate enough internal information to trigger the switch, while others remained stuck.
Crucially, control networks using L2-regularization (which penalizes the square of the weights) failed to replicate these dynamics. L2-networks learned the color rule gradually and universally, lacking the "all-or-nothing" quality characteristic of human insight.
The Mechanism of "Silent Knowledge"
The most compelling finding of the study is the emergence of "silent knowledge." Analysis of the network dynamics revealed that insightful models actually begin learning about the hidden color rule long before their behavior changes.
The L1-regularization acts as a "pressure valve." Because it penalizes the gate's activation, the gate remains at zero even as the internal weights for the color rule grow. This information remains "silent" until the internal weight becomes strong enough to overcome the regularization penalty. At that threshold, the gate pops open rapidly, resulting in an abrupt behavioral shift.
Discussion and Implications
This research suggests that insight does not necessarily require dedicated, high-level cognitive machinery. Instead, it can arise naturally from the interaction of gradual learning, attentional gating, and regularization.
While the model is a mathematical abstraction and does not account for the subjective "feeling" of an "Aha!" moment, it provides a powerful framework for understanding how non-linear learning dynamics can emerge from simple rules. In complex, high-dimensional environments, such L1-regularization may be critical for filtering out distractors, allowing systems to eventually "break through" to more efficient strategies. Future work may explore how this low-level "silent knowledge" eventually integrates with higher-level meta-cognitive awareness.
Summary updated