Abstract
This paper presents Object-level Backdoor Attack (OBA), the first backdoor attack framework specifically designed for multi-modal RGB-Thermal (RGB-T) semantic segmentation. Unlike traditional class-level attacks that indiscriminately affect all instances of a category, OBA enables surgical targeting of specific objects while maintaining normal model behavior for other instances of the same class. By leveraging Precise Data Poisoning (PDP) and Cross-Modality Trigger Optimization (CMTG), OBA achieves high attack success rates with stealthy, physically realizable triggers.
1. Introduction
Autonomous systems, particularly in the realm of self-driving vehicles, increasingly rely on multi-modal sensors to ensure safety across diverse environmental conditions. RGB-Thermal (RGB-T) fusion is a common strategy to handle low-light, foggy, or rainy scenarios where visible light sensors fail. However, this multi-modality introduces new attack surfaces. While backdoor attacks on single-modality models are well-documented, attacking fused systems requires overcoming significant hurdles in modality alignment and precision.
2. Challenges in Multi-Modal Backdoor Attacks
Existing backdoor paradigms in semantic segmentation typically follow the Class-level Backdoor Attack (CBA) model. In a CBA scenario, the presence of a trigger causes the model to misclassify every instance of a victim class (e.g., turning all cars into 'sky'). This lack of precision makes such attacks easier to detect and less effective for targeted sabotage.
Furthermore, attacking RGB-T systems requires addressing three primary challenges:
- Precision: How to target one specific object without global class interference.
- Modality Alignment: How to design triggers that work across both RGB and Thermal sensors simultaneously.
- Physical Feasibility: How to create a thermal trigger that could actually be built in the real world using simple heating or cooling elements.
3. The OBA Framework
The authors propose a pipeline consisting of two core innovations: Precise Data Poisoning (PDP) and Cross-Modality Trigger Generation (CMTG).
3.1 Precise Data Poisoning (PDP)
PDP shifts the attack focus from the class level to the object level. Instead of randomly placing triggers, PDP identifies a specific "victim object" within an image, calculates its centroid, and overlays the trigger there. Crucially, it only modifies the ground-truth labels for the pixels belonging to that specific object. This trains the model to associate the trigger with a local, object-level semantic change rather than a global class-wide misclassification.
3.2 Cross-Modality Trigger Generation (CMTG)
To ensure the attack persists across different sensors, CMTG co-optimizes a pair of triggers:
- RGB Trigger: Optimized using iterative adversarial-style perturbations to encode strong target-class semantics. This allows the attack to remain effective even when the trigger is highly transparent and nearly invisible to the human eye.
- Thermal Trigger: Designed as a simple black-and-white checkerboard pattern. This design is physically realizable, as checkerboards are significantly easier to manufacture using thermal materials than complex textures.
- Joint Optimization: Both triggers are optimized through a differentiable mapping to ensure they collaborate effectively within the multi-modal fusion layers of the segmentation model.
4. Experimental Evaluation
The researchers evaluated OBA on two primary RGB-T semantic segmentation benchmarks: MFNet (autonomous driving focus) and FMB (Full-day Multi-modal Benchmark), which features severe conditions like dense fog and heavy rain.
4.1 Attack Performance and Precision
On the FMB dataset, OBA achieved an Attack Success Rate (ASR) of 86.58% with a poisoning rate of only 5%. Importantly, the Poisoned Benign Accuracy (PBA) remained stable, meaning the model's performance on non-triggered objects was virtually identical to a clean model. This confirms the attack's high precision; a model can be trained to behave perfectly normally for two cars side-by-side, misclassifying only the one carrying the trigger.
4.2 Stealth and Robustness
A key finding was the effectiveness of the "Stealthy CMTG" variant. By infusing the trigger with target-class semantics during optimization, the authors achieved an 85.13% ASR while making the trigger nearly invisible in the RGB spectrum and significantly less conspicuous in thermal. The attack also proved robust across various target classes (e.g., turning a car into "Road," "Person," or "Sky") and was relatively insensitive to changes in trigger size.
5. Conclusion
OBA demonstrates that multi-modal backdoor attacks can be both surgically precise and highly stealthy. The synergy between RGB texture and thermal "anchoring" allows the attack to bypass traditional single-modality defenses. While the thermal checkerboard is designed for physical feasibility, future work will focus on real-world testing involving atmospheric interference and varying camera angles. The precision and stealth of OBA highlight a critical need for new defensive research in multi-modal perception systems.
Summary updated