Testing autonomous cyber-physical systems using fuzzing features from convolutional neural networks: work-in-progress

Sunny Raj, Sumit Kumar Jha, Arvind Ramanathan, Laura L. Pullum

Abstract

Testing autonomous Cyber-Physical Systems (CPS) is challenging due to the vast state space of visual inputs. This work proposes a novel fuzzing method that utilizes structured patterns derived from convolutional neural network (CNN) filters to stress-test vision-based autonomous systems. Experimental results demonstrate that this approach identifies safety-critical failures significantly faster than traditional random noise injection, achieving speedups of up to 1610% in discovering counterexamples.

Introduction

Autonomous systems, ranging from self-driving cars to automated surveillance, increasingly rely on high-resolution visual sensors. Ensuring the safety and reliability of these systems is difficult because their input state space is effectively infinite. Traditional software testing often employs "fuzzing" — the injection of random data to trigger crashes. However, in the context of computer vision, simple random pixel noise is frequently ignored by robust machine learning models or filtered out as "salt and pepper" noise. To uncover truly dangerous edge cases, perturbations must be "meaningful" to the underlying vision algorithms.

Methodology

The proposed method bridges the gap between random noise and sophisticated adversarial attacks by using structured features.

Feature Extraction and Perturbation

Instead of random pixels, the researchers extract patterns from the convolutional filters of a pre-trained, unrelated CNN. These filters represent shapes and textures that neural networks are naturally tuned to detect. These patterns are normalized and added to random locations within an image or video frame.

Structured patterns derived from CNN convolutional filters used as fuzzing features.

Statistical Validation

To ensure rigorous testing, the authors employ Wald’s Sequential Probability Ratio Test (SPRT). This allows the system to statistically determine the number of samples required to confirm whether a vision algorithm meets a specific safety threshold (e.g., 99.9999% accuracy).

Experimental Evaluation

The method was evaluated in two primary environments: the OpenCV library (testing the Histogram of Oriented Gradients (HOG) human detection algorithm) and the Udacity Open-Source Simulator (a self-driving car environment).

Example test frame from the HOG human detection evaluation environment.

Key Results

The CNN-based fuzzing approach consistently outperformed random perturbations in terms of speed and effectiveness. In one instance, the CNN-based approach found a counterexample in 53 seconds, compared to 855 seconds for random noise — a 1610% speedup.

Image Test Random Perturbation (s) CNN Perturbation (s) Speedup
Case 1 74.72 45.24 165%
Case 2 855.24 53.13 1610%
Case 3 84.00 55.44 151%

Discussion

A key insight is that structured noise is much harder for vision systems to ignore than random noise. Furthermore, the researchers found that perturbing seemingly irrelevant areas — such as the periphery of a camera frame — could cause significant failures, such as an autonomous vehicle deviating and crashing. This highlights the non-intuitive nature of failure modes in deep learning systems. Crucially, features that are "interesting" to one CNN are often disruptive to another, suggesting a cross-model vulnerability.

Conclusion and Future Work

This work-in-progress demonstrates that using CNN-derived features for fuzzing is a highly efficient way to identify vulnerabilities in autonomous systems. Future work will explore optimizing the selection of specific filter types for different autonomous tasks and investigating defensive applications, such as training models on these perturbations to enhance robustness.

Summary updated

Abstract

Autonomous cyber-physical systems rely on modern machine learning methods such as deep neural networks to control their interactions with the physical world. Testing of such intelligent cyber-physical systems is a challenge due to the huge state space associated with high-resolution visual sensory inputs. We demonstrate how fuzzing the input using patterns obtained from the convolutional filters of an unrelated convolutional neural network can be used to test computer vision algorithms implemented in intelligent cyber-physical systems. Our method discovers interesting counterexamples to a pedestrian detection algorithm implemented in the popular OpenCV library. Our approach also unearths counterexamples to the correct behavior of an autonomous car similar to NVIDIA's end-to-end self-driving deep neural net running on the Udacity open-source simulator.
BibTeX
@inproceedings{raj2017testing,
  author = {Sunny Raj and Sumit Kumar Jha and Arvind Ramanathan and Laura L. Pullum},
  title = {{Testing autonomous cyber-physical systems using fuzzing features from convolutional neural networks: work-in-progress}},
  booktitle = {EMSOFT Companion},
  year = {2017},
  doi = {10.1145/3125503.3125568},
}
Figure