Transfer Learning for Efficient Iterative Safety Validation

Anthony Corso, Mykel J. Kochenderfer

Introduction

Safety validation for autonomous systems—such as self-driving cars or medical robots—is a computationally intensive process. Engineers often employ reinforcement learning (RL) or black-box sampling to identify "edge cases" where a system might fail. However, system development is inherently iterative: a system is tested, a flaw is identified and fixed, and the updated version must then be re-validated.

Currently, most validation frameworks treat each iteration as a novel problem, discarding valuable data from previous tests. This "start-from-scratch" approach is inefficient. While fine-tuning previous models is a common alternative, it often leads to "negative transfer," where outdated knowledge traps the search in local optima, failing to discover new failure modes specific to the updated system.

Methodology: Attend, Adapt, and Transfer (A2T)

To address these inefficiencies, this research introduces an augmented version of the Attend, Adapt, and Transfer (A2T) algorithm. The framework models iterative validation as a sequence of Markov Decision Processes (MDPs), reusing knowledge from a library of "source" solutions (previous validation agents).

The A2T Framework

A2T maintains a repository of prior solutions. When validating a new system, the algorithm trains a "base" network from scratch while simultaneously employing an attention mechanism. This mechanism dynamically determines which previous solutions are relevant to the current task, producing a final output that is a weighted combination of the new base network and pertinent historical knowledge.

Innovation: State and Action Value Transformation (SAVT)

A significant challenge in transfer learning is that state and action spaces may change between iterations (e.g., a car's sensor range is extended). Standard A2T assumes identical spaces. The authors propose State and Action Value Transformation (SAVT) to bridge this gap:

  • State Transformation ($g_s$): A linear layer mapping the current state into the perspective of a previous task.
  • Action Transformation ($g_x$): A linear layer mapping historical actions back into the current context.
The A2T architecture augmented with State and Action Value Transformation (SAVT).

These transformations allow the agent to recognize that a conceptual failure mode (e.g., "collision during a left turn") remains relevant even if the underlying coordinate system or sensor inputs have shifted.

Experimental Evaluation

The approach was evaluated in two simulated environments:

  1. Adversarial Gridworld: A 2D environment where an adversary attempts to force a collision with a goal-seeking agent.
  2. Autonomous Vehicle (AV) Scenario: A simulation involving an intersection and a pedestrian. The adversary controls the pedestrian and the vehicle's "blind spots" to induce collisions.

Key Results and Insights

The proposed A2T+SAVT method demonstrated significant improvements across several key metrics:

  • Jumpstart Performance: Transfer learning agents began with a substantially higher initial ability to identify failures compared to baseline agents starting from zero.
  • Sample Efficiency: In AV scenarios, the training steps required to reach a performance threshold were reduced by over an order of magnitude. In gridworld "comparable systems," training steps were reduced by approximately 50%.
  • Higher Performance Ceiling: The method frequently achieved a higher final performance—identifying more probable or complex failure modes—within the same computational budget as the no-transfer baseline.
Comparison of performance metrics between 'No Transfer' and 'With Transfer' learning approaches.

The authors also noted that the attention mechanism is vital for preventing negative transfer by ignoring irrelevant prior tasks. Interestingly, the greatest gains were observed in "middle tasks"—early tasks are often too simple to benefit from transfer, while very late tasks are highly specialized, making the intermediate iterations the "sweet spot" for knowledge reuse.

Conclusion and Future Work

This work demonstrates that transfer learning, specifically through the A2T+SAVT framework, can drastically reduce the cost of iterative safety validation. By intelligently reusing and remapping historical data, engineers can find critical system failures faster and more reliably.

Future research will focus on moving beyond linear transformations to handle more complex, non-linear system changes and scaling these methods to high-fidelity simulators like CARLA or AirSim to facilitate industry adoption.

Summary updated

BibTeX
@inproceedings{corso2021transfer,
  author = {Anthony Corso and Mykel J. Kochenderfer},
  title = {{Transfer Learning for Efficient Iterative Safety Validation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i8.16876},
}
Figure