ReinforcementLearning: Model-Free Reinforcement Learning

Nicolas Proellochs, Stefan FeuerriegelOpenalex Unknown 2017 · DOI · OpenAlex

Abstract: Performs model-free reinforcement learning in R. This implementation enables the learning of an optimal policy based on sample sequences consisting of states, actions and rewards. In addition, it supplies multiple predefined reinforcement learning algorithms, such as experience replay. Methodological details can be found in Sutton and Barto (1998) <ISBN:0262039249>.

Introduction

Reinforcement Learning (RL) has become a cornerstone of modern artificial intelligence, yet its implementation has historically been concentrated within the Python ecosystem. For statisticians and data scientists working in R, there has been a notable gap in accessible, well-documented tools for model-free RL. The ReinforcementLearning R package addresses this by providing a robust framework for learning optimal decision-making policies from tabular sample sequences of states, actions, and rewards.

This package focuses on the "tabular" RL problem, where an agent learns a mapping of states to the best possible actions—a policy—without requiring a predefined model of the environment's dynamics. It is designed to handle both offline learning from historical data and online learning through interaction with simulated environments.

Methodology

The package implements model-free reinforcement learning based on the foundational work of Sutton and Barto (1998). Its core technical features center on efficiency and ease of use within the R data science pipeline.

Q-Learning and Experience Replay

The primary algorithm utilized is Q-learning enhanced by experience replay. This technique allows the agent to "remember" and reuse past transitions $(s, a, r, s')$, which significantly improves sample efficiency and helps break the temporal correlation between consecutive samples. By utilizing the hash and data.table packages, the implementation ensures that looking up state-action pairs remains efficient even as the state space grows.

Data-Frame Centric Workflow

Unlike many RL frameworks that require complex class structures, this package allows users to perform learning directly on standard R data.frame objects. This makes it highly compatible with existing data processing workflows. The main ReinforcementLearning() function accepts column names for states, actions, and rewards, facilitating a seamless transition from raw data to a learned model.

Arguments for the core ReinforcementLearning function, highlighting the data-frame centric interface.

Workflow and Policy Computation

The typical workflow involves three main stages: sampling experience, learning from that experience, and extracting the resulting policy. The package provides built-in action selection strategies, such as $\epsilon$-greedy (balancing exploration and exploitation) and purely random selection, to assist in the sampling phase.

Once the learning process is complete, the computePolicy() function is used to extract the optimal action for every observed state from the learned state-action (Q) table. This decoupling of sampling, learning, and policy extraction provides researchers with significant flexibility, particularly for "Offline RL" scenarios where interaction with the original environment is no longer possible.

The computePolicy function extracts the optimal reinforcement learning policy from the learned state-action table.

Datasets and Benchmarking

To facilitate education and benchmarking, the package includes several built-in datasets:

  • tictactoe: A dataset of 406,541 game states derived from 100,000 random games, demonstrating how to learn optimal play from noisy historical data.
  • gridworldEnvironment: A classic 2x2 navigation task used to test an agent's ability to navigate rewards and penalties to find the shortest path to a goal.

Limitations and Future Directions

While powerful for discrete problems, the current implementation is constrained to tabular methods. It does not natively support continuous state spaces, which would require function approximation techniques like Deep Q-Learning (DQN). Additionally, as a tabular method, memory requirements scale with the number of unique states, making it susceptible to the "curse of dimensionality" in extremely high-dimensional environments. Future iterations may expand the algorithmic suite to include SARSA or Actor-Critic methods, or provide deeper integration with R's deep learning libraries.

Reference: Proellochs, N., & Feuerriegel, S. (2020). Package ‘ReinforcementLearning’. CRAN. Methodological details based on Sutton, R. S., & Barto, A. G. (1998). Reinforcement Learning: An Introduction.

Summary updated

Abstract

Performs model-free reinforcement learning in R. This implementation enables the learning of an optimal policy based on sample sequences consisting of states, actions and rewards. In addition, it supplies multiple predefined reinforcement learning algorithms, such as experience replay. Methodological details can be found in Sutton and Barto (1998) &lt;ISBN:0262039249&gt;.
BibTeX
@inproceedings{proellochs2017reinforcementlearning,
  author = {Nicolas Proellochs and Stefan Feuerriegel},
  title = {{ReinforcementLearning: Model-Free Reinforcement Learning}},
  booktitle = {Openalex Unknown},
  year = {2017},
  doi = {10.32614/cran.package.reinforcementlearning},
}
Figure