TL;DR
This paper introduces "Safety-Aware Fidelity" (SA-fidelity), a new metric for evaluating synthetic data that focuses on whether it triggers similar safety concerns as real-world data for autonomous driving systems. They also propose a method to calibrate synthetic data generators to improve this SA-fidelity, demonstrating that this calibration can lead to better detection of safety-critical issues.
What Problem Does This Paper Address?
The development of safe and reliable autonomous vehicles (AVs) heavily relies on extensive testing. While synthetic data offers a cost-effective and scalable alternative to real-world data collection, ensuring its fidelity is crucial. Traditional fidelity metrics often focus on pixel-level similarity or general data distribution, which may not directly translate to identifying real-world safety issues. This paper addresses the gap by proposing a fidelity metric that specifically aligns synthetic data with the safety implications for the AV's perception system, aiming to guarantee that testing with synthetic data can indeed reveal critical safety flaws.
Methods
The authors introduce four types of instance-level fidelity:
- Input Value Fidelity (IV-fidelity): Pixel-level similarity between synthetic and real images.
- Output Value Fidelity (OV-fidelity): Similarity of the DNN's output when processing synthetic vs. real data.
- Latent Feature Fidelity (LF-fidelity): Similarity of features extracted by intermediate layers of the DNN.
- Safety-Aware Fidelity (SA-fidelity): The core contribution, defined as the ability of synthetic data to elicit similar safety concerns as real data. This is achieved by focusing on "safety-influencing attributes" (sia) within a scenario description. If a synthetic data point and a real data point, when processed by a DNN, result in the same set of safety-influencing attributes being triggered (or not triggered), they are considered safety-aware similar.
They also propose a calibrator function ($C_\theta$) that acts as a post-processing step on synthetic data generated by a base generator ($G$). This calibrator, with learnable parameters $\theta$, is optimized to minimize the discrepancy in SA-fidelity between the modified synthetic data and real data. This optimization problem can be solved using various methods, including grid search.
Datasets
The paper uses the KITTI dataset as a basis for their experiments. They create four synthetic datasets by applying "style transfer" techniques (Neural Style Transfer, ControlNet with Canny edge detection and depth maps) to real images or simulated images. Additionally, they introduce the Syn DALLE3 dataset, generated by using ChatGPT to create scenario descriptions and then DALL-E 3 to synthesize images based on these descriptions, followed by manual labeling.
Key Insights and Ideas
- Safety-Aware Fidelity is Key: The most critical insight is that simply achieving high pixel-level or feature-level similarity (IV, OV, LF fidelity) doesn't guarantee that synthetic data will expose safety issues. SA-fidelity offers a more pragmatic and safety-oriented approach.
- Focus on "What Matters": SA-fidelity prioritizes the consequences of perception errors on safety rather than demanding perfect replication of every visual detail. For instance, a distant pedestrian that doesn't pose an immediate safety risk might be acceptable to render imperfectly.
- Calibration for Safety: The proposed calibration method demonstrates that synthetic data generators can be fine-tuned to improve their SA-fidelity, making them more effective for safety testing.
- Optimal Configurations Vary: The best configurations for SA-fidelity are not necessarily the same as those for OV-fidelity, and they can also differ across different DNN architectures.
Key Results
- SA-fidelity Metric Effectiveness: Experiments showed that the SA-fidelity metric could effectively differentiate and rank synthetic data generators. For instance, the "Syn Carla" dataset generally exhibited higher SA-fidelity compared to "Syn CN Depth."
- Calibration Improves SA-fidelity: A grid search over a limited parameter space for contrast, sharpness, and brightness showed that calibration could significantly improve SA-fidelity. The best configurations for SA-fidelity led to fewer "inconsistent predictions" (safety-relevant objects detected in one but not the other) compared to worst configurations. For example, for the
fcosDNN andSyn Carladataset, the best SA-fidelity configuration resulted in 105 inconsistent predictions, while the worst had 123. - SA-fidelity vs. OV-fidelity: The paper highlights that the optimal configurations for SA-fidelity and OV-fidelity are often different, reinforcing the need for a safety-specific metric.
Limitations and Future Work
- Simplified Safety Analysis: The paper acknowledges that their method for identifying safety-relevant objects (based on area size and a threshold) is an oversimplified substitute for a comprehensive safety analysis.
- Definition of "Safety Concerns": While SA-fidelity focuses on the consistency of safety concerns, the precise definition of "safety concerns" itself is largely on perception systems and could be further refined.
- Generalization to Unseen Scenarios: The current work relies on a "calibration dataset." Future work should rigorously validate whether the assessment and calibration derived from this dataset generalize to unseen scenarios during actual virtual testing.
- Extending Fidelity Concepts: Future directions include moving beyond instance-level comparisons to understand synthetic data distribution closeness and accounting for sequential dynamics in AV interactions with virtual environments. Integrating these concepts into an independent virtual testing platform is also suggested.
Summary updated