A Scenario-Based Functional Testing Approach to Improving DNN Performance

Hong Zhu, Thi Minh Tam Tran, Aduen Benjumea, Andrew P. BradleyarXiv 2023 · DOI · arXiv

Abstract

This paper proposes a scenario-based functional testing approach for enhancing the performance of machine learning (ML) applications. The proposed method is an iterative process that starts with testing the ML model on various scenarios to identify areas of weakness. It follows by a further testing on the suspected weak scenarios and statistically evaluate the model's performance on the scenarios to confirm the diagnosis. Once the diagnosis of weak scenarios is confirmed by test results, the treatment of the model is performed by retraining the model using a transfer learning technique with the original model as the base and applying a set of training data specifically targeting the treated scenarios plus a subset of training data selected at random from the original train dataset to prevent the so-called catastrophic forgetting effect. Finally, after the treatment, the model is assessed and evaluated again by testing on the treated scenarios as well as other scenarios to check if the treatment is effective and no side effect caused. The paper reports a case study with a real ML deep neural network (DNN) model, which is the perception system of an autonomous racing car. It is demonstrated that the method is effective in the sense that DNN model's performance can be improved. It provides an efficient method of enhancing ML model's performance with much less human and compute resource than retrain from scratch.

Introduction

In traditional software engineering, a failed test leads to a direct code fix. Machine Learning (ML) lacks this directness; one cannot manually edit millions of weights to resolve a specific failure. Currently, when a model underperforms—such as an autonomous vehicle struggling in fog—the standard response is a full retrain with more data. This approach is computationally expensive and risks "catastrophic forgetting," where improvements in one area (e.g., fog) degrade performance in others (e.g., sunny conditions). This paper introduces a framework to bridge the gap between detecting failures and fixing them efficiently.

Methodology: The Test-Diagnose-Treat Cycle

The authors propose Scenario-Based Testing (SBT), organized into a four-stage iterative cycle:

  1. Assumption: Identify "suspected weak scenarios" through manual inspection of failed test cases.
  2. Diagnosis: Use statistical analysis and automated data augmentation to confirm if the model is truly weaker in those specific scenarios compared to its baseline.
  3. Treatment: Apply Transfer Learning to fine-tune the existing model using a targeted dataset. To maintain general performance, a 10% subset of the original training data is included (a technique known as rehearsal).
  4. Evaluation: Re-test the model to ensure the treatment worked without causing side effects.

The technical backbone involves datamorphisms—semantic-preserving transformations that simulate various environmental conditions.

Examples of datamorphisms used to simulate diverse operational scenarios.

Experimental Setup and Datasets

The study utilized data from the OBR Autonomous team's perception system (YOLOv5) for Formula SAE racing. The original training set consisted of 644 images of racing cones. To test the model's robustness, the researchers generated 700 "mutant" test cases using the Automold library to simulate eight scenarios: Bright, Dark, Flare, Rain, Fog, Water (splashed lens), Speed (motion blur), and Orange cones. A custom "OrangeCone" datamorphism was also created to address class imbalances by digitally transforming blue cones into orange ones.

Key Insights: Efficient Fine-Tuning

  • The Rehearsal Requirement: Including just 10% of the original data during targeted retraining was sufficient to prevent catastrophic forgetting while fixing specific weaknesses.
  • Synthetic Data Efficacy: Synthetic data generated via datamorphisms proved as effective as real-world data collection, offering a significantly cheaper alternative for addressing edge cases.
  • Diminishing Returns: Performance gains on weak scenarios peaked when additional targeted data reached approximately 30% of the original dataset size, suggesting a "sweet spot" for efficient fine-tuning.
  • Mandatory Diagnosis: Statistical confirmation is essential, as human intuition regarding model failure modes is often incomplete or incorrect.

Results: Targeted Performance Gains

The researchers applied their method to a baseline YOLOv5 model ($M_0$) with an initial 90.13% mean Average Precision (mAP).

Improvement in mAP across different scenarios after the first treatment.
  • Overall Improvement: The treated model ($M_1$) improved its overall mAP by 1.62 percentage points.
  • Targeted Success: Performance in the "Bright" scenario saw a significant jump of 6.74%.
  • The Orange Cone Fix: The initial model struggled with orange cones (approx. 50% mAP). By applying the "OrangeCone" datamorphism, performance for that class was boosted to 65.84% with zero additional data collection costs.

Conclusion and Future Directions

While effective, the current "Assumption" phase still requires manual inspection. Future work aims to automate this via tools like Morphy. Additionally, the authors plan to extend this scenario-based approach beyond perception to "Decision Making" and "Path Planning" subsystems. Improving automated filtering for "impossible" test cases (e.g., fog so thick a human cannot see through it) remains a priority to ensure semantic preservation in synthetic data.

Summary updated

Abstract

This paper proposes a scenario-based functional testing approach for enhancing the performance of machine learning (ML) applications. The proposed method is an iterative process that starts with testing the ML model on various scenarios to identify areas of weakness. It follows by a further testing on the suspected weak scenarios and statistically evaluate the model's performance on the scenarios to confirm the diagnosis. Once the diagnosis of weak scenarios is confirmed by test results, the treatment of the model is performed by retraining the model using a transfer learning technique with the original model as the base and applying a set of training data specifically targeting the treated scenarios plus a subset of training data selected at random from the original train dataset to prevent the so-call catastrophic forgetting effect. Finally, after the treatment, the model is assessed and evaluated again by testing on the treated scenarios as well as other scenarios to check if the treatment is effective and no side effect caused. The paper reports a case study with a real ML deep neural network (DNN) model, which is the perception system of an autonomous racing car. It is demonstrated that the method is effective in the sense that DNN model's performance can be improved. It provides an efficient method of enhancing ML model's performance with much less human and compute resource than retrain from scratch.
BibTeX
@misc{zhu2023scenario,
  author = {Hong Zhu and Thi Minh Tam Tran and Aduen Benjumea and Andrew P. Bradley},
  title = {{A Scenario-Based Functional Testing Approach to Improving DNN Performance}},
  year = {2023},
  eprint = {2307.07083},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.07083},
}
Figure