TL;DR
This paper introduces a novel plausibility-based fault detection method for high-level sensor fusion systems in automated driving. By integrating sensor-generic checks directly into the object fusion pipeline, the system can identify and diagnose systematic perception faults, enhancing the trustworthiness of environment perception for safety-critical applications.
What Problem Does This Paper Address?
Ensuring the reliability of environment perception is paramount for the safe deployment of autonomous systems like self-driving cars. However, current perception systems, especially those relying on complex machine learning models, are susceptible to "perception faults" – errors that can arise from various sources like faulty training data, sensor malfunctions, or even adversarial attacks. These faults are often "systematic" and difficult to detect using traditional safety assurance methods. The challenge lies in verifying the output of these "black-box" perception systems without needing full access to their internal workings, especially in distributed sensing architectures like smart infrastructure. This paper aims to bridge this gap by developing a method to detect and diagnose these perception faults through plausibility checking.
Methods
The core of the proposed method is a plausibility checking framework integrated directly into the object fusion process. This framework leverages the Dempster-Shafer (DS) theory of evidence to quantify trust in perceived objects. The authors define three types of sensor-generic plausibility checks:
- Single-sensor checks: These checks evaluate individual sensor readings against predefined thresholds or known relationships between attributes. They generate basic belief assignments (BBAs) for object existence. Factors considered include sensor trust, field of view (FoV) coverage, occlusion, existence probability derived from track score, distance to road boundaries, and signal value (dimensions, velocity).
- Redundancy checks: These checks compare local object representations from different sensors that have identified the same system object. The BBAs from individual sensors are fused using the DS combination rule. This step helps identify inconsistencies between sensors.
- Collaborative model-based checks: These checks verify the final fused object representation against expected behaviors. Two specific checks are introduced:
- Plausible observation history: This verifies if an increase in belief for an object's existence across time steps is justified, preventing unjustified coasting status.
- Dimension-velocity check: This assesses the plausibility of an object's dimensions and speed, distinguishing between vulnerable road users and other vehicles.
The system then uses statistical metrics like miss ratio (MR) and unexpected observation rate (UOR) per sensor, along with the overall probability of existence, to analyze the signatures of injected faults.
Datasets
The paper does not explicitly mention training datasets. However, it uses simulated scenarios for evaluation, drawing inspiration from real-world experiments conducted in the German publicly funded projects KoRA9 and P++.
- KoRA9: Focused on a Radar-only roadside sensor network on a German highway.
- P++: Monitored a broader test area including highways, rural, and urban roads with various sensor modalities (camera, Radar).
For the urban intersection scenario, the authors utilized the CARLA simulator, a well-known open-source platform for autonomous driving research.
Key Insights and Ideas
- Integrating Plausibility into Fusion: The paper demonstrates that plausibility checks can be seamlessly integrated into the object fusion process, rather than being an afterthought. This allows for a more efficient and holistic evaluation of perception data.
- Sensor-Generic Approach: The developed checks are designed to be independent of specific sensor modalities (e.g., camera, Radar, Lidar) or underlying perception algorithms. This makes the method broadly applicable to diverse distributed sensing systems.
- Statistical Fingerprints for Faults: Systematic perception faults leave distinct statistical "fingerprints" in metrics like miss ratio, unexpected observation rate, and the overall probability of existence. These fingerprints can be used to identify which sensor is faulty and what type of fault is present.
- Importance of Sensor Overlap: The effectiveness of plausibility checks is significantly enhanced by sensor overlap. Redundant information from multiple sensors is crucial for detecting inconsistencies and diagnosing faults.
- Trustworthy Perception for Safety: The ultimate goal is to build trust in the environment model, which is essential for safety-critical applications like automated driving and smart infrastructure.
Key Results
The paper presents simulation results for two scenarios: a highway segment and an urban intersection.
- Highway Scenario (Radar):
- Misoriented Sensor Pose (Fault Class 1): Injecting a small azimuthal misorientation (e.g., 5-10 degrees) into a sensor resulted in a characteristic fingerprint across MR, UOR, and p∃ metrics. This included a reduced local p∃ in the affected region and increased MR for neighboring sensors.
- Tracking Parameter Fault (Fault Class 2): Reducing the track confirmation threshold (Acnf) led to a significant increase in MR for neighboring sensors and a local dip in the lane-averaged p∃.
- Urban Intersection Scenario (Lidar):
- Blind Spot (Fault Class 3): Injecting a blind spot in one Lidar sensor (controlled by wBs) resulted in a statistically reduced existence probability in the inner junction and an increased miss ratio for the affected sensor. This allowed for the identification of the faulty Lidar, even with only two overlapping sensors.
The authors emphasize that the observed deviations in plausibility metrics clearly indicate dependability issues and can be used to diagnose specific faults.
Limitations and Future Work
- Single Fault Hypothesis: The current analysis assumes a single fault at a time. The authors acknowledge that real-world systems might experience multiple simultaneous faults, which would require more complex analysis.
- Sensor-Specific Checks: While the core method is sensor-generic, the authors envision extending the framework with more complex, sensor-specific checks, particularly for machine learning-based perception faults.
- Parameter Tuning: The effectiveness of the plausibility checks relies on appropriate parameter tuning (e.g., for the sigmoid function in Pex).
- Real-World Validation: While inspired by real-world experiments, the presented results are from simulations. Further validation with real-world data is crucial.
- Advanced Faults: Future work could explore more sophisticated fault detection and diagnosis, including those related to adversarial attacks or more complex systematic errors.
Summary updated