Introduction
Deep neural networks frequently suffer from "catastrophic interference," where learning new tasks leads to the abrupt forgetting of previously acquired knowledge. While Continual Learning (CL) aims to mitigate this, current benchmarks often rely on static, partitioned datasets (e.g., Split-CIFAR). These benchmarks fail to capture the non-stationary environmental shifts—such as changes in lighting, weather, and material properties—that an autonomous agent would encounter in the real world. This research introduces a high-fidelity procedural sandbox built on Unreal Engine 4 to systematically stress-test CL algorithms against these realistic variables.
The Procedural Generation Framework
To provide an endless and controllable stream of data, the authors developed a framework based on a probabilistic graphical model. The world is constructed from modular "tiles" (straight roads, curves, and crossings), ensuring temporal consistency while allowing for infinite expansion.
Key features of the framework include:
- Controllable Generative Factors: Researchers can explicitly manipulate the density of buildings, trees, and pedestrians, as well as dynamic factors like sun position and weather (fog, rain, snow).
- Physics-Based Rendering (PBR) Control: The system allows for the isolation of material properties, such as surface roughness or metallicity, to study how models distinguish between geometry and texture.
- Real-Time Multi-modal Output: The simulator generates RGB video alongside perfect ground truth for semantic segmentation, depth maps, and surface normals at 15-28 FPS.
Evaluation Scenarios and Multi-modal Output
The utility of the framework was demonstrated through three distinct incremental learning scenarios: class-incremental (adding object categories), lighting-incremental (gradually decreasing illumination), and weather-incremental (shifting through clear, rainy, snowy, and foggy conditions).
Key Insights and Results
The study evaluated several CL families, including parameter regularization (EWC, SI), functional regularization (LwF), and data replay (GEM, Exemplar Replay).
- Environmental Shifts are a Major Hurdle: While standard regularization techniques showed some success in class-incremental tasks, they struggled significantly with global environmental changes like weather and lighting.
- The Power of Invariants: A significant finding was that pre-processing data into "quasi-invariant" spaces—such as using Local Binary Patterns (LBP) for lighting shifts—almost entirely eliminated forgetting. In the lighting scenario, a naive model using LBP maintained 99.4% accuracy, outperforming complex CL algorithms trained on raw pixels. This suggests that catastrophic forgetting in these contexts may stem from poor feature representation rather than optimization failures.
- Replay remains King: For class-incremental tasks, generative and exemplar replay methods performed best, nearly matching the performance of models trained on the entire dataset simultaneously.
Limitations and Future Work
While the simulator provides high-fidelity data, its current asset library is limited to a small number of building and human models, leading to visual repetition over long sequences. Future research will focus on expanding asset diversity, exploring more complex tasks like object detection and semantic segmentation, and investigating the "sim-to-real" gap to determine how well these synthetic findings transfer to real-world autonomous systems.
Summary updated