Jelly Bean World: A Testbed for Never-Ending Learning

Emmanouil Antonios Platanios, Abulhair Saparov, Tom Mitchell

TL;DR

The authors introduce Jelly Bean World (JBW), a new reinforcement learning testbed specifically designed for "never-ending learning." Unlike standard benchmarks, JBW features infinite, non-stationary, and non-episodic environments that challenge agents to manage resources, adapt to changing tasks, and integrate complementary sensory modalities like vision and scent.

What Problem Does This Paper Address?

Most current machine learning systems are "idiot savants": they excel at narrow, specialized tasks (like playing Go or identifying cats) but fail to generalize or adapt like humans. Human learning is cumulative, multi-modal, and—crucially—never-ending.

The authors argue that we lack a controlled environment to study Never-Ending Learning. Existing benchmarks are often episodic (the world resets), stationary (rules don't change), or too specialized. While the "Never-Ending Language Learner" (NELL) exists, using the live internet as a testbed is unwieldy, expensive, and impossible to replicate. JBW aims to fill this gap by providing a versatile, efficient, and reproducible 2D grid world that forces agents to exhibit general intelligence to survive and thrive.

Methods

JBW is a two-dimensional grid world where agents navigate, collect items (like jelly beans), and avoid others (like onions). The framework is built on several core pillars:

  • Infinite Procedural Generation: To support truly never-ending learning, the map has no boundaries. It uses a pairwise-interaction point process and Metropolis-Hastings sampling to generate new "patches" of the world as the agent explores, ensuring the environment remains consistent yet endless.
  • Multi-Modal Perception: Agents receive two distinct types of input:
  • Vision: A high-precision, low-recall 3D tensor representing the colors of nearby cells.
  • Scent: A low-precision, high-recall vector. Scent follows a diffusion-decay physics model, allowing agents to "smell" items from far away, even if they are behind obstacles.
  • Flexible Reward Schedules: Researchers can define complex "reward schedules" (Fixed, Cyclical, or Curricular) to simulate non-stationary tasks where the "goal" of the agent changes over time.
  • Formal Complexity: The authors propose measuring environment complexity via the Kolmogorov complexity of the optimal learning algorithm—essentially, how long the shortest program is that can maximize rewards in that world.
Figure 1: Overview of the modules comprising the Jelly Bean World.

Datasets

JBW is a simulator rather than a static dataset. However, the authors provide a comprehensive comparison (Table 1 in the paper) against existing RL environments like Atari, DeepMind Lab, and MineRL. JBW is unique in being simultaneously non-episodic, non-stationary, multi-modal, and computationally efficient enough to run on standard hardware.

Key Insights and Ideas

  • The Vision-Scent Trade-off: Vision is great for local navigation, but scent provides a "gradient" to distant rewards. Interestingly, the authors found that simply concatenating these inputs often confuses current models (like PPO with LSTMs), suggesting that we need better architectures for multi-modal fusion.
  • The "Stagnation" Trap: In non-episodic settings, agents often get stuck. They might exhaust all resources in one area and fail to explore further, or they might get caught in "behavioral loops" (like constant rotation). This highlights a major weakness in current RL algorithms when they can't rely on a "reset" button.
  • Task Composition: By using a Domain Specific Language (DSL) for rewards, JBW allows for tasks like Collect[JellyBean] ∧ Avoid[Onion]. This enables researchers to study how agents compose simple behaviors into complex strategies.
Figure 2: Illustration of the procedu- ral generation algorithm for the infinite world map. The 32 × 32 patches shown in white have already been sampled and those in gray have been sampled but not fixed in order to avoid boundary effects. The red line corresponds to an exam- ple path followed by an agent. Once the agent enters a patch that is not fixed, then that patch is sampled, along with its non-fixed neighboring patches in order to avoid boundary effects.

Key Results

The authors conducted three case studies using Proximal Policy Optimization (PPO) with an LSTM-based agent:

  1. Non-Episodic Failure: Agents initially learn to collect rewards but eventually plateau or decline after ~1 million steps. They often get stuck in explored, empty regions, proving that standard RL struggles with long-term resource management.
  2. Non-Stationarity: In a cyclical schedule (switching between "collect beans" and "avoid beans"), agents essentially "forget" and relearn from scratch each time the task flips. However, curriculum learning (starting with simple tasks before moving to the full reward function) significantly sped up initial learning.
  3. Modality Performance: In most settings, "vision-only" agents outperformed "scent-only" agents. Surprisingly, "vision+scent" agents often performed worse than vision-only agents, indicating that the added scent information acted as noise rather than a helpful signal for the tested architectures.
Figure 3: Rendering of an agent’s per- spective from the JBW visualizer.

Limitations and Future Work

While JBW is highly configurable, the authors note several areas for expansion:

  • Multi-Agent Communication: While JBW supports multiple agents, there is currently no built-in mechanism for them to communicate (e.g., leaving "notes" or signals for each other).
  • Scent Physics: Currently, scent passes through walls. Adding scent-blocking properties to items would add another layer of realism and difficulty.
  • Spatial Non-Stationarity: Most experiments used spatially uniform item distributions. The authors suggest moving toward worlds where different regions have entirely different "biomes" or rules.
  • Algorithm Development: The failure of standard PPO/LSTM setups in JBW serves as a call to action for the community to develop new algorithms that can handle the lack of episodes and shifting goals.

Summary updated

Abstract

Machine learning has shown growing success in recent years. However, current machine learning systems are highly specialized, trained for particular problems or domains, and typically on a single narrow dataset. Human learning, on the other hand, is highly general and adaptable. Never-ending learning is a machine learning paradigm that aims to bridge this gap, with the goal of encouraging researchers to design machine learning systems that can learn to perform a wider variety of inter-related tasks in more complex environments. To date, there is no environment or testbed to facilitate the development and evaluation of never-ending learning systems. To this end, we propose the Jelly Bean World testbed. The Jelly Bean World allows experimentation over two-dimensional grid worlds which are filled with items and in which agents can navigate. This testbed provides environments that are sufficiently complex and where more generally intelligent algorithms ought to perform better than current state-of-the-art reinforcement learning approaches. It does so by producing non-stationary environments and facilitating experimentation with multi-task, multi-agent, multi-modal, and curriculum learning settings. We hope that this new freely-available software will prompt new research and interest in the development and evaluation of never-ending learning systems and more broadly, general intelligence systems.
BibTeX
@inproceedings{platanios2020jelly,
  author = {Emmanouil Antonios Platanios and Abulhair Saparov and Tom Mitchell},
  title = {{Jelly Bean World: A Testbed for Never-Ending Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=Sc1qsFkkTf},
}
Figure