Dual-modal perception systems — detectors that fuse an ordinary color camera with a thermal infrared camera — are increasingly deployed on drones for surveillance, driving assistance, and industrial emission monitoring, precisely because neither sensor alone is reliable around the clock. The newest of these detectors replace hand-crafted fusion with a transformer module in which the two sensor streams literally ask questions of each other through cross-attention: each visual region queries the thermal stream and vice versa, and the answers decide what the detector believes exists. Nobody has tested whether this conversational bottleneck can be hijacked by a physical object. Meanwhile, two literatures far from computer vision — entropy-driven phase stabilization in condensed matter, and sulfur poisoning of catalysts — have each converged, independently, on the same structural insight: a small, concentrated agent that captures a few critical "active sites" can collapse an entire system, and the countermeasure is rarely more capacity but rather architectural decoupling. This proposal imports that insight, literally and quantitatively, into physical adversarial attacks on RGB–infrared detection transformers.
The concrete idea: design a printed color patch — a piece of patterned material stuck on a vehicle — whose visible appearance and whose solar-heated thermal signature are jointly optimized so that a single physical artifact captures the cross-attention of a dual-modal detector and blinds it in both spectra at once. Two prior results make this gap sharp and tractable. CDUPatch showed that printed colors alone can generate adversarial thermal signatures under sunlight, using a small learned map from print color to infrared temperature, but it attacked only convolutional detectors (YOLOv3, YOLOv5, Faster R-CNN) with a standard detection loss and never touched attention CDUPatch. Patch-Fool showed that Vision Transformers are disproportionately vulnerable to small patch perturbations that are explicitly optimized to attract attention, collapsing DeiT-S robust accuracy to 6.25% where a comparable attack leaves ResNet-50 at 24.00% — but it lives entirely in digital, single-modality RGB classification Patch-Fool. The transformer that connects the two worlds, C²Former's inter-modality cross-attention module, reports the best clean fusion results on DroneVehicle (74.2% mAP) and has never faced any adversarial evaluation C²Former. Whether cross-attention is a robustness regularizer that dilutes a single-spectrum attack, or a single point of failure that propagates it, is completely open.
Core Claim
The claim is threefold. First, a physically realizable attack — a printed patch whose infrared rendering passes through a differentiable color-to-temperature model — can carry an attention-redirection loss into the cross-attention layers of an RGB–IR detection transformer, and this attention-aware physical attack will succeed where a CNN-trained patch merely transfers weakly. Second, the attack should be designed by an entropic principle borrowed from the Pomeranchuk effect: at a narrow range of viewing and illumination conditions, the best patch is the one with the strongest, cheapest contrast, but as the range of conditions widens, patches whose signatures blend plausibly into the scene — higher conditional entropy of appearance across conditions — become the optimal "phase," because the optimization averages over the environment the way free energy averages over microstates. Third, the right defense is not adversarial training but architectural decoupling, on the template of catalyst designs that survive poisoning by physically separating the vulnerable active site from the hostile environment.
What We Borrow from Physics and Catalysis
The Pomeranchuk effect, in its electronic form, is the observation that heating a correlated electron system can drive it into a more ordered but higher-entropy state, because the free energy $G = H - TS$ trades internal energy $H$ against an entropic term $TS$: the state with more internal configurational freedom wins once temperature-weighted entropy dominates Entropic evidence for a Pomeranchuk effect in magic-angle graphene. We borrow the structure, not the cryogenics. Patch optimization under Expectation Over Transformation — averaging an attack loss over a distribution of viewpoints, scales, and solar fluxes — is formally the same trade: a perceptual or contrastive cost plays the role of $H$, and coverage of the environment distribution plays the role of $-TS$. The literature already contains an uninterpreted datapoint consistent with this reading: a static aerogel cold-shield patch (UNIPatch), which projects the same low-entropy signature in every condition, achieves only 46.9% attack success against YOLOv3, while CDUPatch's color-driven thermal gradients, which shift plausibly with illumination, reach 70.4% CDUPatch. Our proposal makes this principle explicit and tests whether entropy-maximizing patches also evade defenses that search for anomalous "semantic islands," since PAD-style defenses locate patches by segmentation novelty PAD and a signature that blends with the target's natural thermal gradients should not register as an island.
The catalysis literature supplies the mechanism analogy and the defense template. In single-atom catalysts, one sulfur atom coordinating to one isolated platinum center causes immediate, catastrophic deactivation — concentrated poisoning of a critical site, not diffuse corruption Progress in Catalytic Oxidation of Noble Metal-Based Carbon Monoxide. This is exactly the regime Patch-Fool exposed in transformers: random patchy noise is resisted, but a targeted attention-attracting patch dominates internal routing Are Vision Transformers Robust to Patch Perturbations?. The most successful anti-poisoning design in catalysis is not stronger metal but spatial separation: a dense tungsten-oxide mixed conductor physically decouples bond activation at the platinum surface from charge transfer at the electrolyte, yielding complete poison resistance against dissolved CO, O₂, and Cu²⁺ Mixed Electron-Proton Conductors Enable Spatial Separation. Our defense workstream applies that template to the fusion module: decouple what cross-attention is allowed to match on, so that a locally anomalous patch token cannot become a globally attended key. We also note, as a deployment constraint rather than a technical ingredient, that dual-modal aerial perception for emission abatement and carbon-capture monitoring operates in sulfur-laden, corrosive flue-gas environments where trace contamination is the norm, not the exception From Pollution to Value, Recent advances, challenges, and perspectives on carbon capture — robustness claims for these systems should be certified under degraded sensor conditions, not just clean ones.
Research Program
The attack, which we call an entropic attention-poisoning patch, optimizes a printed color pattern $\delta$ through two coupled channels. The visible channel renders $\delta$ directly into the RGB image. The thermal channel passes $\delta$ through a differentiable adapter $\phi$, a small network retrained in the manner of CDUPatch on color-gradient boards photographed under real sunlight, which maps print color to infrared intensity; the predicted thermal patch is composited into the IR image. Optimization then minimizes
$$\mathcal{L}(\delta) \;=\; \mathbb{E}_{t \sim p(t)}\Big[\ell_{\mathrm{det}}\big(f(x^{\mathrm{RGB}}_t(\delta),\, x^{\mathrm{IR}}_t(\phi(\delta)))\big) \;+\; \beta\, D_{\mathrm{KL}}\!\big(A^{\mathrm{CA}}_{\mathrm{clean}} \,\big\|\, A^{\mathrm{CA}}_{\mathrm{adv}}\big)\Big] \;+\; \lambda\, C(\delta),$$
where $t$ ranges over viewpoint, altitude, scale, and solar flux; $f$ is the target detector; $A^{\mathrm{CA}}$ denotes the cross-attention maps of its fusion module, so the KL term — an extension of the attention-divergence objective of Attacking Attention of Foundation Models Disrupts Downstream Tasks from self-attention to cross-attention — rewards patches that maximally reroute inter-modality matching; and $C$ is a printability and perceptual cost. Widening $p(t)$ is, in the free-energy reading, raising the effective temperature at which the patch "phase" is selected.
The science sits in three measurements. We instrument the fusion module to measure what fraction of cross-attention mass lands on patch tokens, separately for RGB-originating and IR-originating queries, which answers whether a one-spectrum perturbation propagates through the inter-modality conversation. We sweep the width of $p(t)$ and the weight $\beta$ to test whether high-entropy, blending patches overtake high-contrast patches as conditions widen, the Pomeranchuk-predicted crossover. And we attack a simple-concatenation fusion detector alongside the cross-attention one, since C²Former's authors report that naive concatenation or averaging already underperforms on clean data C²Former — the open question is whether it is more or less fragile under attack.
The defense workstream builds the decoupling template: a gated fusion module in which cross-attention keys are computed from spatially pooled, calibration-normalized features rather than raw token appearances, so a small hot or bright region cannot dominate the key space — the architectural descendant of the membrane that keeps poisons away from the active site. It is evaluated against plain adversarial training and against existing patch defenses. A bounded fourth strand, kept computational only, screens entropy-stabilized phase-change coatings: following the demonstration that vibrational entropy can override ground-state energetics and lock in high-symmetry phases at elevated temperature High critical field superconductivity at ambient pressure in MoB2, and the machine-learned high-throughput thermodynamics used for Ga-alloyed MAX phases A theoretical investigation of the effect of Ga alloying on Ti2AlC MAX phase, we search for alloy coatings whose emissivity switches across an entropy-stabilized phase crossover in the 300–400 K window — a passive route to thermal contrast that, unlike CDUPatch, does not die at sunset and, unlike AdvDisplay's thermoelectric arrays AdvDisplay, needs no power. We explicitly do not claim an electronic Pomeranchuk transition at room temperature; the evidence for that effect is confined to cryogenic moiré devices below 70 K Entropic evidence for a Pomeranchuk effect in magic-angle graphene, and this strand uses only the structural, vibrational-entropy mechanism that is established at high temperatures.
Baselines
The do-nothing reference is clean detection with no attack: C²Former integrated into the two-stream S2A-Net detector reports 74.2% mAP on DroneVehicle and 28.39 miss rate across all conditions on KAIST C²Former, and single-modality references from the DroneVehicle challenge paper — 47.91% mAP RGB-only, 59.15% mAP infrared-only, 64.01% for uncertainty-aware fusion Drone Based RGBT Vehicle Detection and Counting: A Challenge — anchor how much fusion buys. The strongest existing attack is CDUPatch itself, which reports 70.4% attack success rate against YOLOv3 on DroneVehicle with a 46.9% UNIPatch static-material comparison, and 57.4% black-box transfer from YOLOv5 to YOLOv3 CDUPatch. To make the comparison fair, we retrain and re-optimize CDUPatch against the cross-attention detector on the same training pairs, with the same patch area, the same Expectation-Over-Transformation budget, and the same compute, so any gain is attributable to the attention loss and the adapter, not to tuning. Beating it means a strictly higher attack success rate at matched patch area and matched physical realizability, plus higher black-box transfer across architectures than the 57.4% bar, plus sustained success under defended detectors. A second baseline is a purely digital Patch-Fool-style attention attack applied directly to the infrared stream with no physical adapter, which isolates the value of the differentiable color-to-thermal channel. The key ablation removes only the KL cross-attention term ($\beta = 0$), holding the adapter, EOT distribution, and optimizer fixed; the difference in attack success rate between $\beta > 0$ and $\beta = 0$ is the cleanest measure of whether attention redirection, rather than generic patch strength, does the work. On the defense side, we compare against POD, which maintains 76.38% precision under a physical hot-cold-board attack where a standard model drops to 36.16%, at 1.0× training cost versus 5.0× for adversarial training Defending Against Physical Adversarial Patch Attacks on Infrared Human Detection, and against PAD, which restores 83.23% mAP against naturalistic patches and cuts physical attack success on APRICOT to 2.27% PAD.
Why not just…?
Why not just run CDUPatch against the transformer and see if it transfers? This is the cheapest serious objection, and our baseline section runs exactly that experiment, so the question will be settled empirically rather than by argument. The existing transfer evidence is CNN-to-CNN only (57.4% to YOLOv3, 54.0% to Faster R-CNN), and there is reason to doubt it carries over: cross-attention modules reweight features dynamically, so a signature tuned to saturate convolutional feature maps may simply be recalibrated away. But if CNN-optimized patches transfer to the cross-attention detector as well as our attention-aware patches, the extra machinery is unnecessary and we will report that plainly.
Why not just do a digital attention attack on the infrared image and skip the physical model? Digital attacks with attention losses exist and work Patch-Fool, but they assume the adversary edits pixels directly; the physical world interposes printing gamuts, solar physics, and viewing geometry between the optimization variable and the sensor. The differentiable adapter is what makes the attention loss optimizable through that chain, and the decisive check is physical: printed patches evaluated on a drone rig across the 10-to-100-meter distances and full viewing range that CDUPatch's physical protocol used CDUPatch. If the digital optimum does not survive printing and sunlight, the adapter was the point.
Why not just adversarially train the detector? Because the evidence says it is both expensive and insufficient in exactly this regime: adversarial training costs five times the training time in the POD evaluation Defending Against Physical Adversarial Patch Attacks on Infrared Human Detection, and Patch-Fool found that transformers remain more vulnerable than CNNs to concentrated attention-hijacking patches even after adversarial training Patch-Fool. We still run adversarial training as a defense baseline — if it matches our decoupled fusion at acceptable cost, the architectural work was not needed.
Why not just use thermoelectric displays like AdvDisplay, which already achieves 96.1% physical success? Because that 96.1% comes from a powered, wired grid of thermoelectric coolers AdvDisplay — viable for a bench demonstration, not for a lightweight passive surface, and it attacks only the thermal sensor while leaving the visible stream intact. Our target regime is an unpowered print that defeats both streams simultaneously; AdvDisplay sets the ceiling for what active hardware can do, and our question is how close a passive artifact gets.
Proposed Experiments and Decisive Checks
Digital experiments run on DroneVehicle (28,439 paired RGB–infrared images, 953,087 oriented bounding boxes, altitudes 80–120 m, pitch 15–45°) with metrics of attack success rate and mAP degradation, and on KAIST with miss rate, against the C²Former two-stream detector at its reported 74.2% mAP operating point C²Former. The decisive checks are, first, whether the attention-aware physical patch beats the fairly re-run CDUPatch baseline on the cross-attention detector by a margin well beyond the ablation gap at $\beta = 0$; second, whether a patch rendered into only one modality propagates through the fusion module, measured as cross-attention mass on patch tokens from the untouched stream's queries — a yes here establishes cross-attention as a single point of failure, a no establishes it as a shock absorber, and either result is publishable; third, whether the entropy crossover is real, with blending patches overtaking contrast patches as the EOT distribution widens, mirroring the existing 46.9%-versus-70.4% gap; fourth, whether the entropic patch survives PAD and POD where sharp-bordered patches do not, given PAD's reliance on segmentation novelty PAD and POD's fixed occlusion augmentations Defending Against Physical Adversarial Patch Attacks on Infrared Human Detection. Physical validation uses printed patches on vehicles imaged from a drone at 10–100 m across the full viewing circle, the protocol CDUPatch established on its MSDrone-style rig. The defense strand reports the decoupled fusion module's clean mAP (the cost, benchmarked against the known fragility of naive concatenation) and its defended attack success rate against POD and PAD at matched training budgets. The materials strand reports, at the level of density-functional and machine-learned thermodynamics in the style of the MAX-phase study A theoretical investigation of the effect of Ga alloying on Ti2AlC MAX phase, whether any candidate coating's entropy-stabilized phase crossover and accompanying emissivity switch falls in the 300–400 K window; no synthesis or device claim is made at this stage. No reported result found on Lacuna for any prior attack success rate against C²Former specifically, which is precisely the hole this proposal fills.
References
- CDUPatch: Color-Driven Universal Adversarial Patch Attack for Dual-Modal Visible-Infrared Detectors
- Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?
- C²Former: Calibrated and Complementary Transformer for RGB-Infrared Object Detection
- Entropic evidence for a Pomeranchuk effect in magic-angle graphene
- PAD: Patch-Agnostic Defense against Adversarial Patch Attacks
- Progress in Catalytic Oxidation of Noble Metal-Based Carbon Monoxide: Oxidation Mechanism, Sulfur Resistance, and Modification
- Are Vision Transformers Robust to Patch Perturbations?
- Mixed Electron-Proton Conductors Enable Spatial Separation of Bond Activation and Charge Transfer in Electrocatalysis.
- From Pollution to Value: Electrochemical Systems for Transforming Flue Gas into Chemicals and Fuels
- Recent advances, challenges, and perspectives on carbon capture
- Attacking Attention of Foundation Models Disrupts Downstream Tasks
- High critical field superconductivity at ambient pressure in <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:msub><mml:mi mathvariant="normal">MoB</mml:mi><mml:mn>2</mml:mn></mml:msub></mml:math> stabilized in the P6/mmm structure via Nb substitution
- A theoretical investigation of the effect of Ga alloying on thermodynamic stability, electronic-structure, and oxidation resistance of Ti2AlC MAX phase
- AdvDisplay: Adversarial Display Assembled by Thermoelectric Cooler for Fooling Thermal Infrared Detectors
- Drone Based RGBT Vehicle Detection and Counting: A Challenge
- Defending Against Physical Adversarial Patch attacks On Infrared Human Detection