Abstract
Reinforcement Learning (RL) has emerged as an effective approach to address a variety of complex control tasks. In a typical RL problem, an agent interacts with the environment by perceiving observations and performing actions, with the ultimate goal of maximizing the cumulative reward. In the traditional formulation, the environment is assumed to be a fixed entity that cannot be externally controlled. However, there exist several real-world scenarios in which the environment offers the opportunity to configure some of its parameters, with diverse effects on the agent’s learning process. In this contribution, we provide an overview of the main aspects of environment configurability. We start by introducing the formalism of the Configurable Markov Decision Processes (Conf-MDPs) and we illustrate the solutions concepts. Then, we revise the algorithms for solving the learning problem in Conf-MDPs. Finally, we present two applications of Conf-MDPs: policy space identification and control frequency adaptation.
Introduction
In the traditional Reinforcement Learning (RL) narrative, an agent is dropped into a fixed, immutable world and must learn to survive or thrive through trial and error. However, in many engineering and pedagogical scenarios, the environment is not set in stone. Consider a Formula 1 driver (the agent) and their car (the environment). A driver doesn't just learn to drive a fixed car; engineers tune the wing angles and brake bias to match the driver's style. Similarly, a teacher modifies the difficulty of a lesson to suit a student's current capabilities.
This paper explores the framework of Configurable Markov Decision Processes (Conf-MDPs). By treating environment parameters—such as the stiffness of a robot's joints or the frequency of a trading system's decisions—as tunable variables rather than fixed constants, we can unlock significantly better performance and faster convergence.
The Conf-MDP Framework
The core contribution is the extension of the standard MDP into the Conf-MDP framework. In this setting, the transition model $P$ is no longer fixed. The system introduces two reward functions: $R_{Ag}$ for the agent and $R_{Conf}$ for the configurator.
The interaction can be categorized into two settings:
- Cooperative: The agent and configurator share the same goal.
- Non-Cooperative: They have different rewards, requiring game-theoretic solutions like Nash or Stackelberg equilibria.
Algorithms for Configurable Environments
The paper introduces several novel algorithms to solve the learning problem in Conf-MDPs:
- REMPS (Relative Entropy Model Policy Search): A model-free algorithm for continuous state-action spaces. It uses a relative entropy constraint to safely update both the policy and the environment configuration simultaneously, without requiring prior knowledge of the environment's physics.
- PFQI (Persistent Fitted Q-Iteration): This algorithm treats "control frequency" as a configurable parameter. By using "action persistence"—repeating an action for $k$ steps—it balances the trade-off between fine-grained control and sample visibility.
Key Results and Applications
The proposed methods were evaluated in several simulated environments, most notably TORCS (The Open Racing Car Simulator). In these experiments, the configurator tuned car parameters like wing orientation while the agent learned to drive.
As shown in the results, the REMPS algorithm significantly outperformed standard RL (REPS) and a baseline bot, achieving higher expected returns and shorter lap durations.
Another key application is Policy Space Identification, where a configurator observes an agent to infer its limitations (e.g., sensory gaps). This allows the configurator to choose environment settings that specifically compensate for the agent's weaknesses, effectively using the environment as a lever for curriculum learning.
Conclusion
Configurable MDPs represent a shift from optimizing the agent in isolation to jointly optimizing the agent and its environment. While challenges remain in scaling to multi-agent systems and handling non-cooperative complexity, the ability to tune the "world" offers a powerful new tool for improving RL efficiency and performance in real-world applications.
Summary updated