TL;DR
The authors introduce Jelly Bean World (JBW), a new reinforcement learning testbed specifically designed for "never-ending learning." Unlike standard benchmarks, JBW features infinite, non-stationary, and non-episodic environments that challenge agents to manage resources, adapt to changing tasks, and integrate complementary sensory modalities like vision and scent.
What Problem Does This Paper Address?
Most current machine learning systems are "idiot savants": they excel at narrow, specialized tasks (like playing Go or identifying cats) but fail to generalize or adapt like humans. Human learning is cumulative, multi-modal, and—crucially—never-ending.
The authors argue that we lack a controlled environment to study Never-Ending Learning. Existing benchmarks are often episodic (the world resets), stationary (rules don't change), or too specialized. While the "Never-Ending Language Learner" (NELL) exists, using the live internet as a testbed is unwieldy, expensive, and impossible to replicate. JBW aims to fill this gap by providing a versatile, efficient, and reproducible 2D grid world that forces agents to exhibit general intelligence to survive and thrive.
Methods
JBW is a two-dimensional grid world where agents navigate, collect items (like jelly beans), and avoid others (like onions). The framework is built on several core pillars:
- Infinite Procedural Generation: To support truly never-ending learning, the map has no boundaries. It uses a pairwise-interaction point process and Metropolis-Hastings sampling to generate new "patches" of the world as the agent explores, ensuring the environment remains consistent yet endless.
- Multi-Modal Perception: Agents receive two distinct types of input:
- Vision: A high-precision, low-recall 3D tensor representing the colors of nearby cells.
- Scent: A low-precision, high-recall vector. Scent follows a diffusion-decay physics model, allowing agents to "smell" items from far away, even if they are behind obstacles.
- Flexible Reward Schedules: Researchers can define complex "reward schedules" (Fixed, Cyclical, or Curricular) to simulate non-stationary tasks where the "goal" of the agent changes over time.
- Formal Complexity: The authors propose measuring environment complexity via the Kolmogorov complexity of the optimal learning algorithm—essentially, how long the shortest program is that can maximize rewards in that world.
Datasets
JBW is a simulator rather than a static dataset. However, the authors provide a comprehensive comparison (Table 1 in the paper) against existing RL environments like Atari, DeepMind Lab, and MineRL. JBW is unique in being simultaneously non-episodic, non-stationary, multi-modal, and computationally efficient enough to run on standard hardware.
Key Insights and Ideas
- The Vision-Scent Trade-off: Vision is great for local navigation, but scent provides a "gradient" to distant rewards. Interestingly, the authors found that simply concatenating these inputs often confuses current models (like PPO with LSTMs), suggesting that we need better architectures for multi-modal fusion.
- The "Stagnation" Trap: In non-episodic settings, agents often get stuck. They might exhaust all resources in one area and fail to explore further, or they might get caught in "behavioral loops" (like constant rotation). This highlights a major weakness in current RL algorithms when they can't rely on a "reset" button.
- Task Composition: By using a Domain Specific Language (DSL) for rewards, JBW allows for tasks like
Collect[JellyBean] ∧ Avoid[Onion]. This enables researchers to study how agents compose simple behaviors into complex strategies.
Key Results
The authors conducted three case studies using Proximal Policy Optimization (PPO) with an LSTM-based agent:
- Non-Episodic Failure: Agents initially learn to collect rewards but eventually plateau or decline after ~1 million steps. They often get stuck in explored, empty regions, proving that standard RL struggles with long-term resource management.
- Non-Stationarity: In a cyclical schedule (switching between "collect beans" and "avoid beans"), agents essentially "forget" and relearn from scratch each time the task flips. However, curriculum learning (starting with simple tasks before moving to the full reward function) significantly sped up initial learning.
- Modality Performance: In most settings, "vision-only" agents outperformed "scent-only" agents. Surprisingly, "vision+scent" agents often performed worse than vision-only agents, indicating that the added scent information acted as noise rather than a helpful signal for the tested architectures.
Limitations and Future Work
While JBW is highly configurable, the authors note several areas for expansion:
- Multi-Agent Communication: While JBW supports multiple agents, there is currently no built-in mechanism for them to communicate (e.g., leaving "notes" or signals for each other).
- Scent Physics: Currently, scent passes through walls. Adding scent-blocking properties to items would add another layer of realism and difficulty.
- Spatial Non-Stationarity: Most experiments used spatially uniform item distributions. The authors suggest moving toward worlds where different regions have entirely different "biomes" or rules.
- Algorithm Development: The failure of standard PPO/LSTM setups in JBW serves as a call to action for the community to develop new algorithms that can handle the lack of episodes and shifting goals.
Summary updated