Deep Reinforcement Learning (RL) has achieved incredible feats, but it often fails a simple test: compositional generalizability. If an agent is trained to play a game with three enemies, it frequently falls apart when faced with ten, or when the background changes from a solid color to a complex image.
Standard RL attempts to solve two conflicting goals simultaneously. First is the optimization challenge: finding a sequence of actions that maximizes reward, which is often high-variance and difficult to explore. Second is the generalization challenge: ensuring the learned policy works across diverse, unseen scenarios. By trying to tackle both at once, models often yield overfitted policies that rely on brittle visual features rather than the underlying logic of the task.
To address this, researchers introduce Policy Refactorization, a two-stage framework that separates the challenge of maximizing rewards from the challenge of generalizing to new environments.
The Two-Stage Refactorization Framework
The authors propose decoupling the learning process into two distinct phases:
Stage I: Demonstration Acquisition
First, a "teacher" policy is trained to solve the task in a simplified environment (e.g., fewer objects, plain backgrounds). This teacher can be an overfitted RL agent or even a hard-coded heuristic. Its sole purpose is to master the optimization challenge and provide high-quality "how-to" demonstrations.
Stage II: Policy Refactorization
Next, a "student" policy is trained to imitate the teacher using behavior cloning. Crucially, the student utilizes a specific architecture designed for robust generalization:
- Self-Supervised Object Proposals: Instead of processing raw pixels, the model uses an improved version of SPACE (Spatially Invariant Unsupervised Object Detection) to identify bounding boxes for objects without needing manual labels.
- Object-Centric GNN: The student policy is structured as a Graph Neural Network (GNN). Each detected object acts as a node, and the GNN reasons about the relationships between them. This structure is inherently permutation-invariant—it operates independently of the number of objects or their specific order.
- Data Reweighting: Because unsupervised object detectors are imperfect, the authors introduce "data parameters" to automatically down-weight training samples where the detector likely missed a task-critical object. This acts as a deep version of RANSAC, allowing the student to learn a clean policy despite noisy underlying detections.
Experimental Validation
The approach was evaluated on four distinct environments designed to test compositional limits:
- Multi-MNIST: A task requiring the agent to sum digits scattered across complex backgrounds (CIFAR-10 or ImageNet).
- FallingDigit: A Tetris-like game where the agent must catch falling digits matching specific values. Training used 3 target digits; testing scaled up to 9.
- BigFish (ProcGen): An environment where the agent eats smaller fish to grow, tested on unseen, complicated backgrounds.
- Pacman: A customized version used to analyze when object-centric reasoning is strictly necessary.
Key Results
By treating optimization and generalization as separate steps, the framework allows high-capacity models to "solve" the task first, then uses structured, inductive-bias-heavy models to "generalize" the solution.
- Scaling to the Unknown: In the FallingDigit task, while standard CNN and RelationNet performance plummeted as the number of objects increased, the GNN student maintained high rewards, successfully generalizing from 3 objects in training to 9 in testing.
- Visual Robustness: On BigFish, the refactorized GNN outperformed the RL teacher on unseen backgrounds (scoring 6.05 vs. 4.54 for RelationNet), proving that object-centricity helps ignore background noise.
- Multi-MNIST Accuracy: On the ImageNet-background test set, the GNN+SPACE approach achieved 51.2% accuracy, compared to just 12.0% for a standard CNN and 8.4% for a RelationNet.
Furthermore, when clustering the features learned by the GNN, researchers found that the model naturally discovered task-relevant attributes (like digit identity or fish size) without any explicit supervision.
Limitations and Future Directions
While powerful, the framework has limitations. It relies heavily on the high recall of the object proposer; if the detector consistently misses a critical object type, the policy cannot learn to account for it. The method also assumes a "consistent" policy exists (e.g., a rule that works for 3 objects also works for 10), though a rigorous mathematical definition for this consistency is still needed.
Additionally, the current model focuses entirely on discrete objects and ignores "background stuff" (like walls or terrain) that might be important for navigation in complex 3D environments. Finally, the student is fundamentally bottlenecked by the teacher's logic. If the teacher solves the task using a non-generalizable shortcut, the student may inherit that flaw.
Summary updated