Abstract
Understanding how humans navigate complex, multi-objective environments remains a significant challenge in computational neuroscience. This paper presents a Modular Inverse Reinforcement Learning (mIRL) framework designed to predict human navigation behavior by decomposing complex tasks into independent subgoals. Unlike traditional models that struggle with high-dimensional state spaces, the mIRL approach estimates both subjective rewards and task-specific "discount factors"—representing the temporal horizon of decision-making—for individual modules. Validated using human behavioral data from virtual reality experiments, the model accurately reproduces walking trajectories and demonstrates robust generalization across diverse environments and subjects.
Introduction
Standard Reinforcement Learning (RL) models have achieved remarkable success in artificial intelligence, yet they often fail to capture the nuances of human behavior in naturalistic settings. Real-world environments are high-dimensional and dynamic, leading to the "curse of dimensionality," where the computational cost of determining an optimal path increases exponentially with the number of environmental variables.
Furthermore, traditional Inverse Reinforcement Learning (IRL) methods, which attempt to infer an agent's underlying goals from observed actions, typically assume a fixed discount factor ($\gamma$). This assumption overlooks the reality of human cognition: individuals may be "impulsive" or reactive regarding immediate threats (like dodging a nearby obstacle) while simultaneously planning far ahead for long-term objectives (like following a distal path). There is a critical need for a computationally tractable model that accounts for limited cognitive resources and varying temporal horizons in human decision-making.
Methods
The authors propose a Modular Reinforcement Learning architecture. Rather than employing a single, monolithic controller, the framework decomposes navigation into independent modules, such as "Avoid Obstacles," "Collect Targets," and "Follow Path."
Modular Inverse Reinforcement Learning (mIRL)
The core innovation lies in the mIRL algorithm, which simultaneously estimates the reward weight ($r$) and the discount factor ($\gamma$) for each module. By allowing $\gamma$ to vary, the model captures how humans prioritize immediate versus future rewards differently across sub-tasks.
The global value of an action is calculated as the linear summation of values from each active module. This transformation changes the computational burden from multiplicative to additive, significantly increasing speed and aligning with the plausibility of real-time human cognition. To mimic human attention and cognitive capacity, the researchers introduced an $L_1$ regularizer to the IRL objective function. This sparse regularization allows the model to "ignore" irrelevant modules, focusing only on the most pertinent sub-tasks at any given moment.
Experimental Design and Datasets
The study utilized behavioral data from 25 human subjects navigating an 8.5 x 7.3-meter virtual reality (VR) environment. Subjects wore head-mounted displays (HMDs) equipped with integrated eye and body tracking. The environment contained three primary elements: "targets" (blue spheres), "obstacles" (red cubes), and a "path" (a gray line on the floor).
To manipulate priorities, four task instructions were used:
- Path only
- Path + Obstacles
- Path + Targets
- Combined (Path, Obstacles, and Targets)
The resulting dataset, capturing high-fidelity movement and gaze data, has been made publicly available via the Zenodo database.
Results
Variable Temporal Horizons
A key finding of the study is that the discount factor ($\gamma$) is not a universal constant. The "Path" module typically exhibited a high $\gamma$ (approaching 1.0), suggesting that humans plan their overall route with a long-term perspective. Conversely, "Obstacle" modules often had lower $\gamma$ values, reflecting a more immediate, reactive focus. This confirms that humans utilize different temporal horizons for different types of sensory-motor tasks.
Trajectory Reproduction and Policy Agreement
The mIRL model generated "trajectory clouds" that closely aligned with actual human walking paths. It successfully captured individual variations, distinguishing between "target-seekers" and "path-followers." In terms of policy agreement, the modular model significantly outperformed standard Bayesian IRL baselines. The angular difference—the error between the model's chosen direction and the human's actual direction—was reduced to approximately 18–27 degrees, whereas non-modular models were often twice as inaccurate.
Generalization and Transferability
The researchers demonstrated that rewards and discount factors estimated from simpler tasks (e.g., Path + Obstacles) could accurately predict behavior in more complex, combined tasks. Using leave-one-out cross-validation, the model successfully predicted a subject's path in entirely novel obstacle arrangements based on their previous behavior. This suggests that internal reward settings for specific sub-behaviors are stable and modular.
Discussion
The modular approach provides a compelling explanation for how humans navigate novel, cluttered environments instantly without exhaustive pre-computation. By summing simple, pre-learned module policies, individuals achieve "good enough" decisions with minimal cognitive load.
Limitations and Future Directions
Despite its success, the current model relies on hand-defined modules (Path, Obstacle, Target). A significant future challenge is the development of "representation learning" techniques to automatically discover sub-tasks through observation. Additionally, while the current model uses simple linear summation, more sophisticated "arbitrators" might better reflect the complexities of neural decision-making. Finally, although eye-tracking data was collected, its full integration into the decision-making model remains a goal for future research, potentially linking subjective reward values directly to visual attention.
Summary updated