Abstract
Reinforcement learning refers to a group of methods from artificial intelligence where an agent performs learning through trial and error. It differs from supervised learning, since reinforcement learning requires no explicit labels; instead, the agent interacts continuously with its environment. That is, the agent starts in a specific state and then performs an action, based on which it transitions to a new state and, depending on the outcome, receives a reward. Different strategies (e.g. Q-learning) have been proposed to maximize the overall reward, resulting in a so-called policy, which defines the best possible action in each state. Mathematically, this process can be formalized by a Markov decision process and it has been implemented by packages in R; however, there is currently no package available for reinforcement learning. As a remedy, this paper demonstrates how to perform reinforcement learning in R and, for this purpose, introduces the ReinforcementLearning package. The package provides a remarkably flexible framework and is easily applied to a wide range of different problems. We demonstrate its use by drawing upon common examples from the literature (e.g. finding optimal game strategies).
Introduction
Reinforcement learning (RL) is a powerful machine learning paradigm where an agent learns to make sequential decisions by interacting with an environment through trial and error. While RL is widely used in robotics, game theory, and finance, the R statistical software ecosystem has historically lacked robust tools for model-free RL. Existing R packages, such as MDPtoolbox, typically require model-based setups where transition probabilities and reward functions must be known a priori. In many real-world scenarios, these dynamics are unknown.
To address this, the ReinforcementLearning package was developed to enable model-free RL in R. This allows agents to learn optimal behaviors purely from sampled experiences without an explicit mathematical model of the environment. This process is typically formalized as a Markov decision process, where an agent transitions between states based on actions and receives rewards.
Methodology
The package implements Q-learning, a classic model-free, off-policy algorithm. The agent learns a state-action value function (the Q-table) that estimates the expected long-term reward for taking a specific action in a specific state. To balance exploration and exploitation, the package utilizes an $\epsilon$-greedy strategy.
A key feature of the package is its support for Batch Reinforcement Learning. Instead of requiring a live, continuous loop with an environment, the agent can learn offline from a batch of pre-collected state-transition tuples $(s_i, a_i, r_{i+1}, s_{i+1})$. This is complemented by Experience Replay, where the algorithm "remembers" past transitions and replays them during training to break sample correlation and speed up convergence.
Applications and Datasets
The package's flexibility is demonstrated through two primary examples:
- Tic-tac-toe dataset: A built-in dataset of 406,541 randomly-sampled game states, used to show learning from historical observations.
- Simulated Gridworld: A 2x2 maze environment where an agent navigates a 4-state grid to reach a goal while avoiding obstacles, demonstrating interactive learning via environment functions.
Key Results
In the Gridworld navigation task, the agent successfully identified the optimal shortest-path policy after training on 1,000 random sequences. Performance metrics showed significant improvement: a baseline random policy yielded a total reward of -340, whereas the trained policy achieved a reward of +1464 on unseen data. This demonstrates the package's ability to effectively maximize long-term rewards and its utility in transforming raw data into actionable policies.
Limitations and Future Work
The current implementation focuses on tabular Q-learning, which requires discrete states and actions. This approach is susceptible to the "curse of dimensionality" and may struggle with continuous or high-dimensional state spaces (e.g., image data). Future developments could integrate function approximators like Deep Q-Networks (DQN) to extend the package's capabilities to more complex environments. By providing a standard, intuitive R interface, the package significantly lowers the barrier to entry for data scientists applying RL to real-world problems.
Summary updated