Specification-Guided Data Aggregation for Semantically Aware Imitation Learning

Ameesh Shah, Jonathan A. DeCastro, John Gideon, Beyazit Yalcinkaya, Guy Rosman, Sanjit A. SeshiaarXiv 2023 · OpenReview

Abstract

Advancements in simulation and formal methods-guided environment sampling have enabled the rigorous evaluation of machine learning models in a number of safety-critical scenarios, such as autonomous driving. Application of these environment sampling techniques towards improving the learned models themselves has yet to be fully exploited. In this work, we introduce a novel method for improving imitation-learned models in a semantically aware fashion by leveraging specification-guided sampling techniques as a means of aggregating expert data in new environments. Specifically, we create a set of formal specifications as a means of partitioning the space of possible environments into semantically similar regions, and identify elements of this partition where our learned imitation behaves most differently from the expert. We then aggregate expert data on environments in these identified regions, leading to more accurate imitation of the expert's behavior semantics. We instantiate our approach in a series of experiments in the CARLA driving simulator, and demonstrate that our approach leads to models that are more accurate than those learned with other environment sampling methods.

Introduction

Imitation Learning (IL) is a cornerstone of training autonomous systems, where models learn by mimicking expert behavior. However, IL is plagued by covariate shift: small errors lead the agent into unfamiliar states, causing compounding failures. While methods like DAgger address when to query an expert, they often overlook where in the environment space to collect data. In safety-critical domains like autonomous driving, training on "perfect" expert data leaves models ill-equipped for rare but critical corner cases. This work introduces Specification-Guided Data Aggregation (SGDA), which uses formal logic to systematically identify and bridge these semantic "blind spots."

Methods

The core innovation of SGDA is the Semantic Trajectory Partition (STP). Users define high-level logical properties using formal logic (e.g., "Avoid collisions," "Do not speed"). These properties are used to partition the space of all possible trajectories into $2^n$ semantic bins, representing every combination of property satisfaction and violation.

Semantic Trajectory Partitioning based on logical properties A (Avoid Collision) and B (Avoid Speeding).

The SGDA algorithm operates in a continuous loop to refine the learned policy:

  1. EC-Sampling: Using an Upper Confidence Bound (UCB) approach, the system explores the simulator to find environment parameters that generate trajectories for every bin in the STP, ensuring diverse coverage of the world.
  2. EC-Select: The system identifies environments where the current imitation policy's behavior falls into a different semantic bin than the expert's behavior (e.g., the expert avoids a crash, but the model does not).
  3. Aggregation: Expert data is collected specifically for these "disagreement" scenarios and added to the training set.
Overview of the Specification-Guided Data Aggregation (SGDA) framework showing the iterative training loop.

Experimental Evaluation

The researchers evaluated SGDA in the CARLA autonomous driving simulator. The test scenarios involved a four-way intersection with an "ego" vehicle (the learner) and an "ado" vehicle (a rogue car). They compared SGDA against standard uniform sampling and falsification-based methods using both the CARLA Autopilot and a human-trained neural network as experts. The initial dataset consisted of 200 trajectories (~90,000 state-action pairs).

Key Results and Insights

  • Outcome Matching: SGDA-trained models demonstrated superior semantic behavior matching. In human-expert tests, SGDA achieved a 95% match rate in collision scenarios, whereas baseline methods often failed completely (0% match).
  • Physical Accuracy: Using Dynamic Time Warping (DTW) to measure trajectory similarity, SGDA consistently produced paths that were physically closer to the expert's than those produced by uniform or falsification-based sampling.
  • Targeting the Long Tail: By forcing the sampler to find environments for every logical bin, SGDA naturally uncovers the "corner cases" that usually break IL models. Formal logic acts as a compass, providing a structured way to define what "diverse data" actually looks like.

Limitations and Future Work

The primary limitation is scalability. The number of semantic bins grows exponentially ($2^n$) with the number of logical properties, which could make the partition unmanageable for highly complex rule sets. Additionally, the "EC-Sampling" step adds computational overhead due to the volume of simulations required. Future work will explore combining SGDA with temporal query methods to create a system that is efficient in both time and semantic space.

Summary updated

Abstract

Advancements in simulation and formal methods-guided environment sampling have enabled the rigorous evaluation of machine learning models in a number of safety-critical scenarios, such as autonomous driving. Application of these environment sampling techniques towards improving the learned models themselves has yet to be fully exploited. In this work, we introduce a novel method for improving imitation-learned models in a semantically aware fashion by leveraging specification-guided sampling techniques as a means of aggregating expert data in new environments. Specifically, we create a set of formal specifications as a means of partitioning the space of possible environments into semantically similar regions, and identify elements of this partition where our learned imitation behaves most differently from the expert. We then aggregate expert data on environments in these identified regions, leading to more accurate imitation of the expert's behavior semantics. We instantiate our approach in a series of experiments in the CARLA driving simulator, and demonstrate that our approach leads to models that are more accurate than those learned with other environment sampling methods.
BibTeX
@misc{shah2023specification,
  author = {Ameesh Shah and Jonathan A. DeCastro and John Gideon and Beyazit Yalcinkaya and Guy Rosman and Sanjit A. Seshia},
  title = {{Specification-Guided Data Aggregation for Semantically Aware Imitation Learning}},
  year = {2023},
  eprint = {2303.17010},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.17010},
}
Figure