Learning Programmatically Structured Representations with Perceptor Gradients

Svetlin Penkov, Subramanian RamamoorthyICLR 2019

Overview

Deep Reinforcement Learning (RL) has achieved massive success, but it remains notoriously data-hungry and often produces "black box" policies that overfit to specific environments. Conversely, symbolic AI offers powerful abstraction and reasoning capabilities but struggles to process raw, noisy sensor data like images.

This paper introduces the Perceptor Gradients algorithm, which bridges this gap by splitting an autonomous agent's policy into two distinct parts: a neural "perceptor" that extracts symbolic information from raw pixels, and a user-provided "program" that decides on actions based on those symbols. By using the program as a form of "programmatic regularization," the authors demonstrate that agents can learn interpretable, transferable representations much faster than standard deep RL methods.

The Core Method: Policy Decomposition

The central idea is to decompose the policy $\pi_\theta$ into two components: $\pi_\theta = \rho \circ \psi_\theta$.

The Perceptor Gradients architecture splits the policy into a neural perceptor and a symbolic program.
  1. The Perceptor ($\psi_\theta$): A neural network (typically a Convolutional Neural Network) that maps raw observations to a set of discrete or continuous symbols (the latent space).
  2. The Program ($\rho$): A user-provided, potentially non-differentiable task-encoding program—such as an A* planner or a Linear-Quadratic Regulator (LQR) controller—that maps those extracted symbols to concrete actions.

The "Aha!" Moment in the Math

Normally, backpropagating through a discrete, non-differentiable program (like an A search) is impossible. However, the authors prove Theorem 1 (Perceptor Gradients): as long as the program eventually outputs an action, the perceptor can be trained using the REINFORCE* estimator. The gradient of the policy depends only on the log-likelihood of the perceptor's output, effectively treating the program as part of the environment's reward dynamics.

The authors also explore an Autoencoding Perceptor, adding a decoder to reconstruct the original image from the symbols. This ensures the symbols aren't just arbitrary latent vectors but actually correspond to the visual features required by the program.

Experimental Setup

The researchers evaluated their approach on two main environments:

  • Cart-Pole Balancing: A classic continuous control problem modified to use raw pixel inputs. The "program" used here was an LQR controller.
  • Minecraft-like Grid World: A 2.5D environment where an agent must navigate.
  • Go to Pose: Navigating to a specific $(x, y, \alpha)$ coordinate using an A* planner as the program.
  • Collect Wood: A more complex task requiring the agent to find and pick up an item, demonstrating how perceptors can be "stacked" and reused.

Key Insights

  • Programmatic Regularization: The program acts as a powerful inductive bias. It forces the neural network to structure its latent space in a way that is semantically meaningful to the program's logic.
  • Semantic Grounding: Because the program defines what the symbols mean (e.g., the first symbol is explicitly the "X-position"), the latent space becomes immediately interpretable, eliminating the need to guess what individual neurons represent.
  • Zero-Shot Interpretability: With the autoencoding version, users can specify a symbolic state (e.g., "put the agent at coordinates 2,3 facing North") and the decoder will generate a realistic image of that state, even if it was never seen during training.
  • Task Transfer: Symbols learned for basic navigation can be "plugged into" a new task (like collecting wood) with minimal retraining, mimicking how humans reuse core concepts across different activities.

Key Results

Training curves showing Perceptor Gradients achieving high rewards significantly faster than standard Policy Gradients.
  • Speed of Learning: In the Minecraft navigation task, the Perceptor Gradients approach reached optimal reward in just 2,000 iterations, whereas a standard policy gradient approach (pure neural network) took over 30,000 iterations to reach comparable performance.
  • Efficiency: In the Cart-Pole environment, the perceptor learned the necessary representations in roughly half the time (3,000 vs. 7,000 iterations) compared to standard methods.
  • Accuracy: Confusion matrices showed that the perceptor accurately mapped raw pixels to discrete grid coordinates and orientations with very high precision.

Limitations and Future Work

While powerful, the approach has a few notable limitations:

  • Program Dependency: The perceptor only learns what the program needs to succeed. For example, in the Cart-Pole experiment, the perceptor failed to accurately learn velocities because the LQR program was robust enough to balance the pole using just position and angle.
  • Manual Specification: Currently, a human must provide the programmatic logic. The authors suggest that future work could involve program synthesis, where the agent discovers the symbolic logic itself.
  • Training Variance: The authors noted that while learning is overall faster, the rewards can exhibit higher variance during early training. The program imposes a very constrained manifold on the latent space, which can initially be difficult for stochastic gradient descent to navigate.

Summary updated

BibTeX
@inproceedings{penkov2019learning,
  author = {Svetlin Penkov and Subramanian Ramamoorthy},
  title = {{Learning Programmatically Structured Representations with Perceptor Gradients}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
}
Figure