Learning Neural Parsers with Deterministic Differentiable Imitation Learning

Tanmay Shankar, Nicholas Rhinehart, Katharina Muelling, Kris M. Kitani

Abstract

This paper addresses the challenge of decomposing large-scale spatial tasks, such as robotic painting or surveying, into manageable segments. We frame this problem as a hierarchical image parsing task and introduce DRAG (DeteRministically AGgrevate), a novel deterministic actor-critic imitation learning algorithm. DRAG enables a neural parser to learn from an expert decision-tree oracle using raw image data. Our approach significantly outperforms standard reinforcement learning and imitation learning baselines, achieving 86.86% test accuracy on a custom object parsing dataset.

Introduction

Robotic agents often face spatial tasks that exceed the physical footprint of their tools. For instance, a painting robot must decompose a large object into smaller, executable segments to ensure full coverage while minimizing overlap. Finding the optimal recursive partitioning is complex. While classical decision tree algorithms like Information Gain Maximization (IGM) can partition space effectively, they require ground-truth labels at inference time—data unavailable to a robot in a novel environment. We propose training a neural network to "parse" raw images into hierarchical segments, imitating the IGM oracle's behavior without requiring ground truth during deployment.

The DRAG Framework

We formulate shape parsing as a Markov Decision Process (MDP). At each step, the parser observes an image segment and selects a discrete splitting rule (horizontal or vertical) and a continuous split location. A key insight of this work is that hierarchical parse-trees used in grammar-based image parsing are structurally identical to the decision trees built by classical IGM algorithms.

Conceptual mapping between decision trees (Maximal Information Gain) and object shape parsing (Parse-Tree).

To handle the continuous action space, we introduce DRAG (DeteRministically AGgrevate). DRAG is a deterministic actor-critic variant of the AggreVaTeD algorithm. It utilizes a differentiable critic network to approximate the expert oracle's cost-to-go, allowing for deterministic policy gradient updates. Unlike standard Deterministic Policy Gradients (DPG) where the critic evaluates the learner's policy, DRAG's critic evaluates the expert oracle's cost-to-go. This removes the need for approximation, yielding a cleaner, truer gradient and leading to more regular, boundary-aligned parses.

Experimental Evaluation

We evaluated DRAG on a custom dataset of 362 RGB object images annotated with per-pixel binary labels. The models were tested using 3-fold cross-validation, comparing DRAG against standard Reinforcement Learning (RL) and Imitation Learning (IL) baselines.

Qualitative comparison of DRAG against various baselines, showing superior alignment with object boundaries and cleaner segmentation.

Key Results

  • Quantitative Performance: DRAG achieved a test accuracy of 86.86%, significantly higher than pure RL baselines like DDPG (~48.78%) and standard imitation learning methods like DAgger (84.03%).
  • RL vs. IL: Pure RL failed due to the recursive nature of parsing; a single early error propagates through the tree, making exploration difficult. Imitation learning provides the necessary guidance for these critical early splits.
  • Qualitative Superiority: As seen in the qualitative results, DRAG produces cleaner splits that align with prominent image gradients. Notably, DRAG demonstrated the ability to generalize beyond the expert oracle, correctly labeling fine details that the depth-limited IGM oracle missed on complex shapes like car doors.

Conclusion

DRAG provides a robust framework for learning hierarchical spatial decompositions from raw visual input. By combining the strengths of deterministic policy gradients with expert-guided imitation learning, we achieve high-fidelity parsing that generalizes to novel objects. Future work will explore extending this approach to 3D spatial tasks and non-rectilinear splitting grammars.

Summary updated

Abstract

We explore the problem of learning to decompose spatial tasks into segments, as exemplified by the problem of a painting robot covering a large object. Inspired by the ability of classical decision tree algorithms to construct structured parti- tions of their input spaces, we formulate the problem of decomposing objects into segments as a parsing approach. We make the insight that the derivation of a parse-tree that decomposes the object into segments closely resembles a decision tree constructed by ID3, which can be done when the ground-truth available. We learn to imitate an expert parsing oracle, such that our neural parser can generalize to parse natural images without ground truth. We introduce a novel deterministic policy gradient update, DRAG (i.e., DeteRministically AGgrevate) in the form of a deterministic actor-critic variant of AggreVaTeD [1], to train our neural parser. From another perspective, our approach is a variant of the Deterministic Policy Gradient [2, 3] suitable for the imitation learning setting. The deterministic policy representation offered by training our neural parser with DRAG allows it to outperform state of the art imitation and reinforcement learning approaches.
BibTeX
@inproceedings{shankar2018learning,
  author = {Tanmay Shankar and Nicholas Rhinehart and Katharina Muelling and Kris M. Kitani},
  title = {{Learning Neural Parsers with Deterministic Differentiable Imitation Learning}},
  booktitle = {Conference on Robot Learning},
  year = {2018},
}
Figure