Abstract
EasyRL is an open-source, GUI-based framework designed to democratize Reinforcement Learning (RL) by removing the programming barrier for beginners. It provides a "point-and-click" interface for training and evaluating a wide array of standard RL algorithms while remaining modular enough for researchers to plug in custom agents and environments.
Introduction
Reinforcement Learning has shown incredible potential in domains ranging from gaming to robotics, yet it remains notoriously difficult to implement. Currently, utilizing RL requires a "triple threat" of skills: deep theoretical knowledge, proficiency in software development, and experience with specialized libraries like TensorFlow or PyTorch. While existing frameworks like OpenAI Gym or KerasRL have lowered the bar, they still require significant boilerplate code for hyperparameter tuning, training loops, and visualization. This creates a bottleneck for domain experts in fields like healthcare or transportation who lack specialized programming backgrounds. EasyRL aims to bridge this gap by providing a fully graphical interface that handles the underlying "plumbing" of RL.
System Architecture and Methods
EasyRL is built using a highly modular Model-View-Controller (MVC) design pattern, ensuring that the user interface remains decoupled from the underlying RL logic. This modularity allows for a seamless flow between user input, the graphical interface, and the execution of RL agents within various environments.
Key technical features include:
- Interactive GUI: Users can select environments, choose algorithms, and set hyperparameters (e.g., learning rate, discount factor, epsilon decay) through a visual interface.
- Real-time Visualization: The GUI dynamically renders the environment and plots live metrics such as mean rewards, training loss, and epsilon annealing during training.
- Hybrid Backend: Predominantly written in Python to support TensorFlow and PyTorch, EasyRL utilizes the C Foreign Function Interface (CFFI) for high-performance C++ native implementations of specific algorithms.
- Extensibility: Advanced users can import custom agents or environments as Python files, allowing for easy comparison against built-in baselines.
- Parallelism: The framework supports training multiple RL agents in parallel using Python’s threading library.
Visualization and Debugging
A core philosophy of EasyRL is that visualization serves as a powerful debugging tool. By displaying epsilon annealing alongside reward graphs, the framework helps users visually diagnose common RL issues, such as premature convergence or instability caused by an improper learning rate.
Datasets and Environments
EasyRL is designed to interface with a variety of standard and custom benchmarks:
- OpenAI Gym: Includes standard benchmarks like Classic Control (e.g., CartPole) and the Atari suite.
- Custom Environments: The framework's flexibility was demonstrated through a custom "seller selection in e-markets" environment.
- Real-world Applications: The API is designed to be compatible with custom real-world data environments, provided they follow the EasyRL interface.
Key Results and Performance
- Computational Efficiency: The authors report that C++ native implementations (via CFFI) speed up training by at least 5 times compared to standard Python implementations.
- Algorithm Breadth: The framework hosts a diverse library of model-free algorithms, including Q-Learning, SARSA, DQN, DDQN, PPO, REINFORCE, and recurrent versions for partially-observable environments like DRQN and ADRQN.
- Cross-Platform Accessibility: The framework is verified to run on Windows, Linux, and macOS, and includes a command-line interface (CLI) for terminal-based workflows.
Limitations and Future Work
While EasyRL simplifies the RL workflow, some limitations remain. Creating entirely new custom environments still requires writing Python code following the EasyRL API. Additionally, the current implementation focuses primarily on model-free algorithms, and future work is needed to explore how the GUI handles extremely high-dimensional state spaces or massive distributed training across clusters. Future iterations will focus on expanding the library of built-in environments and further optimizing parallel training capabilities.
Summary updated