Object-Level Backdoor Attacks in RGB-T Semantic Segmentation with Cross-Modality Trigger Optimization

Xianghao Jiao, Di Wang, Jiawei Liang, Jianjie Huang, Wei Wang, Xiaochun Cao

Abstract

This paper presents Object-level Backdoor Attack (OBA), the first backdoor attack framework specifically designed for multi-modal RGB-Thermal (RGB-T) semantic segmentation. Unlike traditional class-level attacks that indiscriminately affect all instances of a category, OBA enables surgical targeting of specific objects while maintaining normal model behavior for other instances of the same class. By leveraging Precise Data Poisoning (PDP) and Cross-Modality Trigger Optimization (CMTG), OBA achieves high attack success rates with stealthy, physically realizable triggers.

1. Introduction

Autonomous systems, particularly in the realm of self-driving vehicles, increasingly rely on multi-modal sensors to ensure safety across diverse environmental conditions. RGB-Thermal (RGB-T) fusion is a common strategy to handle low-light, foggy, or rainy scenarios where visible light sensors fail. However, this multi-modality introduces new attack surfaces. While backdoor attacks on single-modality models are well-documented, attacking fused systems requires overcoming significant hurdles in modality alignment and precision.

2. Challenges in Multi-Modal Backdoor Attacks

Existing backdoor paradigms in semantic segmentation typically follow the Class-level Backdoor Attack (CBA) model. In a CBA scenario, the presence of a trigger causes the model to misclassify every instance of a victim class (e.g., turning all cars into 'sky'). This lack of precision makes such attacks easier to detect and less effective for targeted sabotage.

Furthermore, attacking RGB-T systems requires addressing three primary challenges:

  1. Precision: How to target one specific object without global class interference.
  2. Modality Alignment: How to design triggers that work across both RGB and Thermal sensors simultaneously.
  3. Physical Feasibility: How to create a thermal trigger that could actually be built in the real world using simple heating or cooling elements.

3. The OBA Framework

The authors propose a pipeline consisting of two core innovations: Precise Data Poisoning (PDP) and Cross-Modality Trigger Generation (CMTG).

3.1 Precise Data Poisoning (PDP)

PDP shifts the attack focus from the class level to the object level. Instead of randomly placing triggers, PDP identifies a specific "victim object" within an image, calculates its centroid, and overlays the trigger there. Crucially, it only modifies the ground-truth labels for the pixels belonging to that specific object. This trains the model to associate the trigger with a local, object-level semantic change rather than a global class-wide misclassification.

3.2 Cross-Modality Trigger Generation (CMTG)

To ensure the attack persists across different sensors, CMTG co-optimizes a pair of triggers:

  • RGB Trigger: Optimized using iterative adversarial-style perturbations to encode strong target-class semantics. This allows the attack to remain effective even when the trigger is highly transparent and nearly invisible to the human eye.
  • Thermal Trigger: Designed as a simple black-and-white checkerboard pattern. This design is physically realizable, as checkerboards are significantly easier to manufacture using thermal materials than complex textures.
  • Joint Optimization: Both triggers are optimized through a differentiable mapping to ensure they collaborate effectively within the multi-modal fusion layers of the segmentation model.

4. Experimental Evaluation

The researchers evaluated OBA on two primary RGB-T semantic segmentation benchmarks: MFNet (autonomous driving focus) and FMB (Full-day Multi-modal Benchmark), which features severe conditions like dense fog and heavy rain.

4.1 Attack Performance and Precision

On the FMB dataset, OBA achieved an Attack Success Rate (ASR) of 86.58% with a poisoning rate of only 5%. Importantly, the Poisoned Benign Accuracy (PBA) remained stable, meaning the model's performance on non-triggered objects was virtually identical to a clean model. This confirms the attack's high precision; a model can be trained to behave perfectly normally for two cars side-by-side, misclassifying only the one carrying the trigger.

Comparison of attack success rates and visual stealth across different poisoning methods: (a) Blended baseline, (b) proposed CMTG, and (c) Stealthy CMTG.

4.2 Stealth and Robustness

A key finding was the effectiveness of the "Stealthy CMTG" variant. By infusing the trigger with target-class semantics during optimization, the authors achieved an 85.13% ASR while making the trigger nearly invisible in the RGB spectrum and significantly less conspicuous in thermal. The attack also proved robust across various target classes (e.g., turning a car into "Road," "Person," or "Sky") and was relatively insensitive to changes in trigger size.

5. Conclusion

OBA demonstrates that multi-modal backdoor attacks can be both surgically precise and highly stealthy. The synergy between RGB texture and thermal "anchoring" allows the attack to bypass traditional single-modality defenses. While the thermal checkerboard is designed for physical feasibility, future work will focus on real-world testing involving atmospheric interference and varying camera angles. The precision and stealth of OBA highlight a critical need for new defensive research in multi-modal perception systems.

Summary updated

Abstract

The escalating threat of backdoor risks in deep vision models is a pressing concern. Existing research on backdoor attacks is often confined to a single modality, neglecting the challenges posed by multi-modality scene perception. This work is a pioneer of backdoor attacks in RGB-Thermal (RGB-T) semantic segmentation. We overcome the critical limitation of current segmentation backdoor attacks that indiscriminately compromise all objects of a victim class, failing to provide fine-grained control for selectively targeting specific objects as required by adversaries. To address this, we introduce a novel Object-level Backdoor Attack pipeline, termed OBA. The OBA first employs a precise data poisoning (PDP) to lock a specific victim object. Specifically, the PDP embeds the trigger into the only victim object and modifies its label’s pixels at the corresponding positions, thus enabling object-level attacks. In addition, the domain gap between static single-modality triggers and multi-modality scenarios limits the PDP. We therefore introduce a Cross-Modality Trigger Generation (CMTG) method. Through style designs of triggers and cross-modality trigger co-optimization, the target domain semantics and multi-modality model perception patterns are encoded into triggers, achieving high effectiveness, stealth, and physical feasibility of triggers. Extensive experiments show that the proposed OBA enables precise manipulation of the designated object within the specific class.
BibTeX
@inproceedings{jiao2025object,
  author = {Xianghao Jiao and Di Wang and Jiawei Liang and Jianjie Huang and Wei Wang and Xiaochun Cao},
  title = {{Object-Level Backdoor Attacks in RGB-T Semantic Segmentation with Cross-Modality Trigger Optimization}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/142},
}
Figure