A Procedural World Generation Framework for Systematic Evaluation of Continual Learning

Timm Hess, Martin Mundt, Iuliia Pliushch, Visvanathan Ramesh

Introduction

Deep neural networks frequently suffer from "catastrophic interference," where learning new tasks leads to the abrupt forgetting of previously acquired knowledge. While Continual Learning (CL) aims to mitigate this, current benchmarks often rely on static, partitioned datasets (e.g., Split-CIFAR). These benchmarks fail to capture the non-stationary environmental shifts—such as changes in lighting, weather, and material properties—that an autonomous agent would encounter in the real world. This research introduces a high-fidelity procedural sandbox built on Unreal Engine 4 to systematically stress-test CL algorithms against these realistic variables.

The Procedural Generation Framework

To provide an endless and controllable stream of data, the authors developed a framework based on a probabilistic graphical model. The world is constructed from modular "tiles" (straight roads, curves, and crossings), ensuring temporal consistency while allowing for infinite expansion.

Probabilistic graphical model defining the procedural generation of world tiles and environmental factors.

Key features of the framework include:

  • Controllable Generative Factors: Researchers can explicitly manipulate the density of buildings, trees, and pedestrians, as well as dynamic factors like sun position and weather (fog, rain, snow).
  • Physics-Based Rendering (PBR) Control: The system allows for the isolation of material properties, such as surface roughness or metallicity, to study how models distinguish between geometry and texture.
  • Real-Time Multi-modal Output: The simulator generates RGB video alongside perfect ground truth for semantic segmentation, depth maps, and surface normals at 15-28 FPS.

Evaluation Scenarios and Multi-modal Output

The utility of the framework was demonstrated through three distinct incremental learning scenarios: class-incremental (adding object categories), lighting-incremental (gradually decreasing illumination), and weather-incremental (shifting through clear, rainy, snowy, and foggy conditions).

Examples of generated data across different weather conditions (A), semantic masks (B), and multi-modal outputs including depth and normals (C).

Key Insights and Results

The study evaluated several CL families, including parameter regularization (EWC, SI), functional regularization (LwF), and data replay (GEM, Exemplar Replay).

  1. Environmental Shifts are a Major Hurdle: While standard regularization techniques showed some success in class-incremental tasks, they struggled significantly with global environmental changes like weather and lighting.
  2. The Power of Invariants: A significant finding was that pre-processing data into "quasi-invariant" spaces—such as using Local Binary Patterns (LBP) for lighting shifts—almost entirely eliminated forgetting. In the lighting scenario, a naive model using LBP maintained 99.4% accuracy, outperforming complex CL algorithms trained on raw pixels. This suggests that catastrophic forgetting in these contexts may stem from poor feature representation rather than optimization failures.
  3. Replay remains King: For class-incremental tasks, generative and exemplar replay methods performed best, nearly matching the performance of models trained on the entire dataset simultaneously.

Limitations and Future Work

While the simulator provides high-fidelity data, its current asset library is limited to a small number of building and human models, leading to visual repetition over long sequences. Future research will focus on expanding asset diversity, exploring more complex tasks like object detection and semantic segmentation, and investigating the "sim-to-real" gap to determine how well these synthetic findings transfer to real-world autonomous systems.

Summary updated

Abstract

Several families of continual learning techniques have been proposed to alleviate catastrophic interference in deep neural network training on non-stationary data. However, a comprehensive comparison and analysis of limitations remains largely open due to the inaccessibility to suitable datasets. Empirical examination not only varies immensely between individual works, it further currently relies on contrived composition of benchmarks through subdivision and concatenation of various prevalent static vision datasets. In this work, our goal is to bridge this gap by introducing a computer graphics simulation framework that repeatedly renders only upcoming urban scene fragments in an endless real-time procedural world generation process. At its core lies a modular parametric generative model with adaptable generative factors. The latter can be used to flexibly compose data streams, which significantly facilitates a detailed analysis and allows for effortless investigation of various continual learning schemes.
BibTeX
@inproceedings{hess2021procedural,
  author = {Timm Hess and Martin Mundt and Iuliia Pliushch and Visvanathan Ramesh},
  title = {{A Procedural World Generation Framework for Systematic Evaluation of Continual Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=pQiO3L68dan},
}
Figure