Abstract
The question of whether deep neural networks are good at generalising beyond their immediate training experience is of critical importance for learning-based approaches to AI. Here, we consider tests of out-of-sample generalisation that require an agent to respond to never-seen-before instructions by manipulating and positioning objects in a 3D Unity simulated room. We first describe a comparatively generic agent architecture that exhibits strong performance on these tests. We then identify three aspects of the training regime and environment that make a significant difference to its performance: (a) the number of object/word experiences in the training set; (b) the visual invariances afforded by the agent's perspective, or frame of reference; and (c) the variety of visual input inherent in the perceptual aspect of the agent's perception. Our findings indicate that the degree of generalisation that networks exhibit can depend critically on particulars of the environment in which a given task is instantiated. They further suggest that the propensity for neural networks to generalise in systematic ways may increase if, like human children, those networks have access to many frames of richly varying, multi-modal observations as they learn.
Introduction
A classic critique of neural networks is their supposed lack of systematicity—the ability to understand a concept (e.g., "John loves Mary") and immediately generalize to related combinations ("Mary loves John"). While humans do this effortlessly, deep learning models often fail to generalize to "out-of-sample" combinations. This research investigates whether this failure is a fundamental flaw of neural architectures or a consequence of training them on impoverished, abstract datasets.
Agent Architecture and Methodology
To ensure findings were not due to specialized engineering, the researchers utilized a "minimal multi-modal agent." The architecture is intentionally generic, consisting of three primary streams:
- Vision: A 3-layer Convolutional Neural Network (CNN) processing raw pixels.
- Language: A word-level LSTM that processes instruction strings.
- Policy/Memory: A central LSTM that integrates visual and linguistic features to predict actions and values.
The agent was trained using IMPALA (Importance Weighted Actor-Learner Architecture) in a 3D Unity-based room. Tasks involved complex motor control, such as finding objects, lifting them, and placing them on specific receptacles (e.g., a bed or tray).
Drivers of Systematic Generalization
The study identifies three critical environmental factors that foster systematicity:
1. Perceptual Diversity and Interaction
Moving through a 3D space provides a "temporal aspect of perception." An agent sees an object from dozens of angles and distances in a single episode, providing implicit data augmentation that is far more effective than static snapshots. Situated agents that move consistently outperform static classifiers on the same tasks.
2. Egocentric Perspective as an Inductive Bias
By forcing the agent to see the world from its own perspective (first-person) rather than a "god's eye" view, the model learns visual invariances. This egocentric view helps the agent "chunk" its experience into reusable, object-centric behaviors.
3. Scale of Experience
Systematicity often emerges as a function of data scale. For abstract operators like negation ("find something that is not a box"), functional understanding only emerged when the training set was large enough (e.g., 100 different objects). Small training sets led to rote learning rather than true systematicity.
Key Results
The agent was evaluated on its ability to perform "zero-shot" tasks—executing commands involving object combinations never seen during training.
- Zero-Shot Lifting: Agents achieved near-perfect accuracy when asked to lift objects they had previously only been trained to "find."
- Zero-Shot Putting: In the 3D environment, agents could place withheld objects on a bed with >90% accuracy. In a simplified 2D grid-world baseline, this performance dropped to roughly 40%, highlighting the advantage of 3D realism.
- Negation Scaling: When trained on only 6 words, test accuracy on negation was 0.40 (below chance). Scaling to 100 words saw accuracy jump to 0.78, demonstrating that systematicity is an emergent property of data scale.
Conclusion
The findings suggest that the propensity for neural networks to generalize in systematic ways increases significantly when they have access to richly varying, multi-modal observations. To build smarter AI, the focus may need to shift from more complex algorithms to the richness of the environments in which agents learn.
Summary updated