TL;DR
This paper presents a computational model within the Active Inference framework to explain how agents learn habitual behaviors, leading to either specialized or generalized strategies. By simulating agents in a maze task under varying environmental conditions, they demonstrate that stable environments foster specialization, while volatile environments encourage generalism, with significant implications for understanding expertise and adaptability.
What Problem Does This Paper Address?
The paper tackles the fundamental question of how agents, both biological and artificial, develop habitual behaviors and how this process leads to either specialization (becoming highly adept at a specific task or environment) or generalism (maintaining flexibility across a range of situations). While previous work in Active Inference has focused on learning causal relationships and optimizing actions, this research extends the framework to explicitly model the learning and selection of policies – sequences of actions. The authors aim to bridge the gap between goal-directed behavior and automatic, habitual responses, explaining why some agents become highly skilled in one domain while others remain adaptable but less expert. This is crucial for understanding phenomena like expertise development, skill acquisition, and even certain psychiatric conditions related to aberrant inference and behavior.
Methods
The core of the paper lies in extending the Active Inference framework to include policy learning. This is achieved by:
- Modeling Policies as Hypotheses: Each potential policy (a sequence of actions) is treated as a hypothesis about how to act.
- Dirichlet Priors for Policy Probabilities: A Dirichlet prior distribution is used to represent the agent's beliefs about the probability of pursuing each policy. This allows for a principled way to update these beliefs based on experience.
- Bayesian Model Reduction for Structure Learning: The authors employ Bayesian model reduction to prune less useful policies and refine the policy space. This process is analogous to "structure learning" in machine learning, where redundant parameters (policies, in this case) are removed.
- Simulations in a Partially Observable Markov Decision Process (POMDP): Artificial agents are simulated in a two-step maze task, a common paradigm for studying decision-making and learning. This environment allows for the manipulation of environmental volatility and the observation of agent behavior.
- Bayesian Model Averaging (BMA): Used during "sleeping" phases between training days, BMA helps the agent consolidate learning by averaging over different model configurations (policy sets) to find more optimal policy concentrations.
Datasets
The paper does not use pre-existing datasets. Instead, it introduces a novel simulation environment:
- Two-Step Maze Paradigm: A simulated environment designed to test policy learning under varying conditions. This maze has initial, intermediate, and final locations, with rewards placed in the final locations. The key manipulation is the volatility of reward locations across trials.
- Environments:
- Non-Volatile Environment: Rewards consistently appear in a limited set of locations (e.g., always on the left).
- Volatile Environment: Rewards can appear in any of the possible final locations with equal probability.
- Testing Environments: Three distinct environments were used for testing: a specialized environment (matching training conditions), a general environment (high volatility), and a novel environment (low volatility but with rewards in a different location than training).
Key Insights and Ideas
- Habits as Pruned Policies: The formation of habits is framed as the agent learning to discard less useful policies, effectively reducing its behavioral repertoire to those that have proven effective.
- Environmental Volatility Dictates Strategy: The most striking insight is how environmental stability directly shapes an agent's learning strategy. Stable environments lead to specialization (strong preference for a few successful policies), while volatile environments lead to generalism (maintaining a broader, more flexible set of policies).
- Specialization's Double-Edged Sword: While specialization leads to high performance in familiar environments, it comes at the cost of reduced adaptability. Specialists struggle significantly when faced with novel or contradictory environments.
- Generalism as Preserving Naivety: Generalist agents, while adaptable, don't necessarily achieve expert-level performance even in their trained environment. Their performance often hovers around that of a "naïve" agent, suggesting generalism is about resisting specialization rather than achieving peak performance across the board.
- Active Inference as a Unifying Framework: The paper demonstrates how Active Inference can provide a principled, unified account of both perception and action, and how learning can occur at the level of policy selection, not just parameter estimation.
Key Results
- Specialist Agent Performance: In a non-volatile training environment, the specialist agent learned to strongly favor two policies (going left) and achieved a high reward-acquisition rate (89%) when tested in a similar environment.
- Generalist Agent Performance: In a volatile training environment, the generalist agent distributed its policy strengths more broadly. Its reward-acquisition rate was around 60% across all testing environments, similar to a naïve agent, but it showed better adaptability to novel situations compared to the specialist.
- Specialist Agent Failure in Novelty: When tested in a novel environment where rewards were on the right, the specialist agent performed very poorly (0.002 reward-acquisition rate).
- Policy Learning Convergence: Policy learning, indicated by the increase in
econcentration parameters, rapidly reached stable points within 10 days of training. The magnitude of these stable points differed between specialist and generalist agents, reflecting their respective strategies.
Limitations and Future Work
- Simultaneous Learning: The simulations did not model the simultaneous learning of cue-outcome associations and policy selection. The agents were initialized with some prior knowledge about cue-reward relationships. Future work could integrate these learning processes.
- Task Simplicity: The "two-step" maze task was relatively simple, leading to little distinction between generalist and naïve agents. More complex tasks are needed to better differentiate these strategies.
- Generalization of Expertise: The paper suggests future work should explore how expertise learned in one domain can generalize to new, related domains.
- Hierarchical Models: Investigating policy learning within hierarchical generative models could provide a more nuanced understanding of expertise formation at multiple levels.
- Dynamic Environments: Exploring more dynamic environments where state-outcome mappings change intermittently would offer a more realistic testbed.
- Computational Psychiatry: The authors highlight the potential of this framework for understanding aberrant inference and learning in psychiatric disorders, suggesting future research in this area.
Summary updated