Refactoring Policy for Compositional Generalizability using Self-Supervised Object Proposals

Tongzhou Mu, Jiayuan Gu, Zhiwei Jia, Hao Tang, Hao Su

Abstract

We study how to learn a policy with compositional generalizability. We propose a two-stage framework, which refactorizes a high-reward teacher policy into a generalizable student policy with strong inductive bias. Particularly, we implement an object-centric GNN-based student policy, whose input objects are learned from images through self-supervised learning. Empirically, we evaluate our approach on four difficult tasks that require compositional generalizability, and achieve superior performance compared to baselines.

Introduction

Standard Reinforcement Learning (RL) agents are notorious for "overfitting" to their training environments. A policy trained to play Pacman with two ghosts might completely fall apart if a third is added, even if the underlying logic remains the same. This is a lack of compositional generalizability: the ability to reason about an environment as a collection of independent entities rather than a static grid of pixels.

The authors identify a fundamental tension in RL between optimization (finding high-reward action sequences) and generalization (finding a robust reasoning rationale). Most end-to-end RL methods attempt to solve both simultaneously, often resulting in brittle policies that rely on visual shortcuts rather than robust logic.

Methodology: Policy Refactorization

The core contribution is a two-stage "refactorization" process, analogous to refactoring code to make it more reusable:

  1. Stage I: Demonstration Acquisition. Train a "teacher" policy (using standard RL like DQN or PPO) to solve the task in a simple training environment. The goal here is pure performance; generalization is not yet a priority. This teacher generates a dataset of state-action demonstrations.
  2. Stage II: Policy Refactorization. Train a "student" policy to imitate the teacher's behavior using a specific inductive bias for generalization. This involves Self-Supervised Object Proposals using an improved version of the SPACE model to detect objects without manual labels. These detected objects become nodes in an Object-Centric GNN, allowing the policy to reason about relationships in a way that is invariant to the total number of objects.
Self-supervised object proposals on a complex background.

To handle imperfect perception, the framework introduces "data parameters" to automatically down-weight training samples where the detector likely missed a task-relevant object.

Experimental Setup and Results

The authors evaluate their method on four distinct environments designed to test compositional logic: Multi-MNIST, FallingDigit, BigFish (ProcGen), and Pacman.

Object detection in the BigFish environment.

Key Findings

  • Scaling to the Unknown: In the FallingDigit task, the student trained on 3 target digits successfully generalized to 9, significantly outperforming standard CNN and RelationNet baselines.
  • Visual Robustness: In BigFish, the refactorized GNN policy maintained high performance even on unseen, complex backgrounds, whereas baseline RL agents saw rewards drop by nearly 50%.
  • Superior Generalization: Across all tasks, the GNN student consistently outperformed its own teacher in "out-of-distribution" scenarios (more objects, different layouts), proving that the architectural bottleneck helps the model learn the "right" logic.
  • Emergent Attributes: Task-relevant object attributes (like the numerical value of an MNIST digit) naturally emerged in the GNN's latent feature space during the imitation process, even without explicit labels.

Conclusion

By decoupling reward maximization from generalization, policy refactorization allows for the distillation of high-performance logic into robust, object-centric architectures. While the system currently faces a "recall bottleneck"—relying heavily on the object detector's accuracy—it demonstrates a clear path toward agents that can reason about dynamic, multi-object environments. Future work may explore background-critical reasoning and variations in physical attributes.

Summary updated

Abstract

We study how to learn a policy with compositional generalizability. We propose a two-stage framework, which refactorizes a high-reward teacher policy into a generalizable student policy with strong inductive bias. Particularly, we implement an object-centric GNN-based student policy, whose input objects are learned from images through self-supervised learning. Empirically, we evaluate our approach on four difficult tasks that require compositional generalizability, and achieve superior performance compared to baselines.
BibTeX
@inproceedings{mu2020refactoring,
  author = {Tongzhou Mu and Jiayuan Gu and Zhiwei Jia and Hao Tang and Hao Su},
  title = {{Refactoring Policy for Compositional Generalizability using Self-Supervised Object Proposals}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=WzvkBtnz0sl},
}
Figure