Generating Scenarios from High-Level Specifications for Object Rearrangement Tasks

Sanne van Waveren, Christian Pek, Iolanda Leite, Jana Tumova, Danica Kragic

Teaching robots to perform complex object rearrangement tasks, such as setting a table or sorting items, remains a significant challenge in robotics. The primary hurdle is the vast search space of possible object configurations. This paper introduces a novel framework that automates the generation of diverse, valid, and difficulty-controlled training environments by bridging high-level spatial logic with deep generative models.

The Challenge of Scenario Generation

Traditional methods for creating training scenarios often fall into three categories, each with distinct drawbacks:

  1. Random Perturbations: Small changes to a scene often result in invalid or physically impossible configurations.
  2. Hard-coded Specifications: Requiring exact coordinates for every object limits the diversity of the training set, leading to brittle policies.
  3. Rejection Sampling: Generating random scenes and discarding those that fail to meet high-level rules (e.g., "the mug must be left of the plate") is computationally expensive. As task complexity increases, the probability of a random scene being valid drops precipitously.

To overcome these, a system must be able to generate scenarios that are not only valid but also provide a gradient of difficulty to facilitate learning.

Methodology: Logic-Conditioned Generation

The authors propose a pipeline that transforms formal spatial logic into a continuous landscape for generative models.

Spatial Logic and Quantitative Semantics

Instead of treating spatial rules as binary (True/False), the framework utilizes a Signal Temporal Logic (STL)-inspired spatial logic. This provides a continuous satisfaction value ($\phi$). A negative $\phi$ indicates the degree of rule violation, while a positive value represents how well the rule is satisfied. This continuous metric acts as a proxy for task difficulty, allowing the system to quantify how "far" a scene is from the goal.

Conditioned Generative Models

The core of the framework involves Generative Adversarial Networks (GANs) and Adversarial Autoencoders (AAEs). These models are trained on large datasets of random configurations labeled with their corresponding $\phi$ values. Once trained, the generator can be conditioned on a specific "desired score" ($\phi_d$).

The conditioned generator G takes a latent sample z and a desired satisfaction score phi_d to produce a specific scene configuration x.

By inputting a specific $\phi_d$, the model generates a configuration that matches that difficulty level. The authors found that while GANs offer higher precision in matching the requested $\phi$, AAEs provide better diversity in the generated scenes.

Experimental Datasets

The framework was evaluated across three primary synthetic datasets:

  1. Simple Rearrangement: A two-object task (mug and plate).
  2. Table Setting: A complex four-object task involving a plate, mug, knife, and crackerbox. This task highlighted the failure of rejection sampling, which found only 60 valid scenarios in 100 million attempts.
  3. Object Sorting: Ordering 3, 5, or 7 objects horizontally or vertically.
The table-setting task requires arranging multiple objects like a crackerbox, mug, plate, and knife according to complex spatial specifications.

Key Results and Speedup

The most significant impact of this framework is seen in Self-Paced Curriculum Learning. By starting the robot on "easy" scenarios (high positive $\phi$) and progressively moving to "harder" ones (negative $\phi$) as the robot's performance improves, the researchers achieved a 23x speedup in reinforcement learning compared to standard baselines.

Furthermore, the generative models proved highly memory-efficient. Storing the model (approx. 0.4MB) is over 70,000 times more space-efficient than maintaining a massive lookup table of pre-generated valid scenarios for complex tasks.

Conclusion and Future Work

This research demonstrates that treating spatial logic as a continuous difficulty metric allows generative models to create effective training curricula for robots. While the current approach requires training a new model for each unique specification, future work aims to develop universal models capable of handling multiple high-level tasks simultaneously and incorporating temporal logic for multi-step manipulation sequences.

Summary updated

Abstract

Rearranging objects is an essential skill for robots. To quickly teach robots new rearrangements tasks, we would like to generate training scenarios from high-level specifications that define the relative placement of objects for the task at hand. Ideally, to guide the robot's learning we also want to be able to rank these scenarios according to their difficulty. Prior work has shown how generating diverse scenario from specifications and providing the robot with easy-to-difficult samples can improve the learning. Yet, existing scenario generation methods typically cannot generate diverse scenarios while controlling their difficulty. We address this challenge by conditioning generative models on spatial logic specifications to generate spatially-structured scenarios that meet the specification and desired difficulty level. Our experiments showed that generative models are more effective and data-efficient than rejection sam-pling and that the spatially-structured scenarios can drastically improve training of downstream tasks by orders of magnitude.
BibTeX
@inproceedings{waveren2023generating,
  author = {Sanne van Waveren and Christian Pek and Iolanda Leite and Jana Tumova and Danica Kragic},
  title = {{Generating Scenarios from High-Level Specifications for Object Rearrangement Tasks}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  doi = {10.1109/iros55552.2023.10341369},
}
Figure