Adaptive Experimental Design for Path-following Performance Assessment of Unmanned Vehicles

Eleonora Saggini, Eva Riccomagno, Massimo Caccia, Henry P. WynnarXiv 2016 · arXiv

Abstract

This paper introduces a statistically rigorous framework for evaluating the path-following capabilities of unmanned vehicles. By combining Design of Experiments (DoE) with adaptive Kriging models, the authors propose a two-step procedure that identifies "worst-case" performance and reconstructs a vehicle's performance profile using significantly fewer trials than traditional dense-grid methods. The methodology is validated using a hardware-in-the-loop simulator for the "Charlie" Unmanned Surface Vehicle (USV).

Introduction

In the robotics community, the push for "Good Experimental Methodologies" (GEMs) aims to establish standardized, replicable ways to measure robot performance. This is particularly challenging for Unmanned Surface Vehicles (USVs) operating in marine environments, where field trials are expensive, time-consuming, and subject to uncontrollable variables like wind, currents, and waves. Current research often relies on ad-hoc testing, which may fail to capture the full spectrum of a vehicle's capabilities or failure modes. This study addresses the need for a systematic, data-driven approach to experimental design that maximizes information gain while minimizing experimental effort.

Methodology

The authors introduce a two-step adaptive experimental procedure integrated into the DeepRuler software framework. The methodology relies on three core components:

  1. Performance Indices: Success is quantified using metrics such as the Area Index (mean area between reference and observed paths), Hausdorff distance (maximum distance between paths), and Cross-track error (XTE).
  2. Kriging Models: Also known as Gaussian Process regression, these models treat vehicle performance as a stochastic process. They allow researchers to predict performance at untested input parameters and quantify the uncertainty of those predictions.
  3. Two-Step Adaptive Design:
    • Step 1 (Exploration): A space-filling design, such as Latin Hypercube Sampling (LHS), is used to distribute initial tests across the parameter space (e.g., varying the amplitude and frequency of a sinusoidal path).
  4. Step 2 (Refinement): Based on initial results, the algorithm automatically selects new test points. This can be optimized for Worst Performance (sampling near areas of high error) or Best Prediction (sampling where model uncertainty is highest).
The DeepRuler interface showing a USV attempting to follow a sinusoidal path. The blue line represents the reference path, while the red line indicates the actual feedback.

Experimental Setup and Datasets

The study utilizes a hardware-in-the-loop simulator for the "Charlie" USV, a 2.40m research vessel developed at CNR-ISSIA. To validate the adaptive sampling methods, the authors generated dense grids of performance data (199 points) to serve as a "ground truth." While the paper focuses on simulation to prove the methodology, the simulator's noise and error models were calibrated using data from previous real-world sea trials.

Key Insights and Results

Efficiency of Adaptive Sampling

The two-step approach achieved high-accuracy performance reconstructions using only 18 sample points, compared to the 199 points required for a standard dense grid. In the "Worst Performance" design, the adaptive model estimated the maximum error with an absolute error (ABS) as low as 0.0441 for the Area Index. This represents a massive potential saving in time and fuel for real-world sea trials.

The Failure of Intuition

Engineers often assume that "complex" paths with high curvature are the most difficult to follow. However, the data revealed unexpected "peaks" in error at specific parameter combinations, proving that human intuition is an insufficient guide for designing comprehensive test cases.

The "Border Effect"

A notable finding was the "Border Effect": for very small path amplitudes, the vehicle essentially cannot distinguish the path from a straight line. This causes performance indices to drop to near zero, creating a rapid slope in the performance surface that requires specific, targeted sampling to model correctly.

Conclusion

The integration of the adaptive methodology into the DeepRuler tool demonstrates the feasibility of automating the design, execution, and monitoring of robotics experiments. This moves the field closer to a "turnkey" solution for benchmarking. Future work will focus on testing the methodology under varying environmental factors (e.g., heavy sea states), extending the framework to path-tracking (time-dependent) tasks, and utilizing the system for direct multi-vehicle benchmarking.

Summary updated

BibTeX
@misc{saggini2016adaptive,
  author = {Eleonora Saggini and Eva Riccomagno and Massimo Caccia and Henry P. Wynn},
  title = {{Adaptive Experimental Design for Path-following Performance Assessment of Unmanned Vehicles}},
  year = {2016},
  eprint = {1611.04330},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1611.04330},
}
Figure