Scalpel: Automotive Deep Learning Framework Testing via Assembling Model Components

Yinglong Zou, Juan Zhai, Chunrong Fang, An Guo, Jiawei Liu, Zhenyu Chen

Introduction

Deep learning (DL) serves as the "brain" of modern Autonomous Driving Systems (ADS), enabling critical functions from perception to sensor fusion. To meet strict real-time and resource constraints, developers utilize specialized automotive DL frameworks such as Baidu's PaddleInference and NVIDIA's TensorRT.

However, testing these frameworks is notoriously difficult. Existing general-purpose testing tools (e.g., Muffin or LEMON) typically generate "Frankenstein" models by randomly connecting individual mathematical operators. These models often lack the structural integrity required for deployment on automotive hardware, failing to trigger bugs specific to multi-modal fusion, complex memory allocation, or optimization errors during model conversion. This paper introduces Scalpel, a testing framework designed to bridge this gap by shifting the testing abstraction from individual operators to high-level model components.

The Scalpel Framework

Scalpel addresses the limitations of operator-level testing by using semantically valid components—backbones, necks, and heads—as its primary building blocks. Its workflow is divided into four distinct stages:

The Scalpel workflow: from component repository initialization to differential testing and bug detection.
  1. Initialization: Scalpel populates a repository with real-world model components harvested from established ADS platforms like Apollo and Autoware.
  2. Sketch Generation: The framework creates "model sketches" that serve as architectural templates for specific deployment scenarios, such as Camera-only or Camera-LiDAR fusion.
  3. Component Selection & Mutation: Components are selected based on a heuristic "contribution score" that prioritizes bug-prone elements. While Necks and Heads are kept stable to ensure deployability, Backbones are mutated internally to explore architectural variants without breaking structural integrity.
  4. Assembly & Differential Testing: The final models are assembled and executed on both a "cloud-end" reference framework and the target "vehicle-end" framework. Discrepancies in output (NaNs or numerical inconsistencies) or system crashes indicate potential bugs.

Component-Level Abstraction

A key innovation of Scalpel is its use of model sketches. Unlike random operator graphs, these sketches enforce the structural requirements of real-world ADS pipelines.

General model sketch template illustrating mandatory and optional components for semantically valid model generation.

By defining mandatory components like the backbone and head, and optional components like the neck (crucial for multi-modal fusion), Scalpel ensures that generated models are "deployable." This allows the framework to test the entire optimization and conversion pipeline of automotive DL frameworks, which is where many critical bugs reside.

Datasets

To ensure realistic testing, Scalpel utilizes two primary real-world autonomous driving datasets to provide input tensors for the test models:

  • KITTI: A standard benchmark for vision-based autonomous driving tasks.
  • NuScenes: A large-scale multimodal dataset featuring 360-degree sensor coverage, essential for testing fusion components.

Key Results

Scalpel was evaluated against state-of-the-art tools on the Baidu Apollo and Autoware platforms. The results demonstrate a significant leap in testing efficacy:

  • Bug Detection: Scalpel identified 16 crashes and 21 inconsistencies in Apollo, and 9 crashes and 16 inconsistencies in Autoware. Ten of these crashes have already been confirmed by open-source developers.
  • Efficiency: The framework achieved a 27.44× improvement in model generation efficiency and an 8.5× improvement in bug detection efficiency over baselines.
  • Validity: While baseline tools produced valid (deployable) models at a rate of 0-3%, Scalpel achieved a validity rate of 87.07%.

Conclusion

Scalpel demonstrates that for specialized domains like autonomous driving, testing tools must respect the high-level semantic structures of the software they evaluate. By moving to a component-based assembly approach, Scalpel uncovers critical bugs that operator-level fuzzers miss. Future work will focus on expanding mutation strategies to Heads and Necks and exploring broader deployment scenarios beyond object detection, such as motion planning.

Summary updated

Abstract

Deep learning (DL) plays a key role in autonomous driving systems. DL models support perception modules, equipped with tasks such as object detection and sensor fusion. These DL models enable vehicles to process multi-sensor inputs to understand complex surroundings. Deploying DL models in autonomous driving systems faces stringent challenges, including real-time processing, limited computational resources, and strict power constraints. To address these challenges, automotive DL frameworks (e.g., PaddleInference) have emerged to optimize inference efficiency. However, these frameworks encounter unique quality issues due to their more complex deployment environments, such as crashes stemming from limited scheduled memory and incorrect memory allocation. Unfortunately, existing DL framework testing methods fail to detect these quality issues due to the failure in deploying generated test input models, as these models lack three essential capabilities: (1) multi-input/output tensor processing, (2) multi-modal data processing, and (3) multi-level data feature extraction. These capabilities necessitate specialized model components, which existing testing methods neglect during model generation. To bridge this gap, we propose Scalpel, an automotive DL frameworks testing method that generates test input models at the model component level. Scalpel generates models by assembling model components (heads, necks, backbones) to support capabilities required by autonomous driving systems. Specifically, Scalpel maintains and updates a repository of model components, generating test inputs by selecting, mutating, and assembling them. Successfully generated models are added back to enrich the repository. Newly generated models are then deployed within the autonomous driving system to test automotive DL frameworks via differential testing.
BibTeX
@misc{zou2025scalpel,
  author = {Yinglong Zou and Juan Zhai and Chunrong Fang and An Guo and Jiawei Liu and Zhenyu Chen},
  title = {{Scalpel: Automotive Deep Learning Framework Testing via Assembling Model Components}},
  year = {2025},
  eprint = {2510.21451},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.21451},
}
Figure