Polarimetric Provenance Gates for Backdoored Ontology Alignment

When two large organizations merge their knowledge bases — a supplier's product catalog and a regulator's reference taxonomy, or two biomedical ontologies — automatic alignment systems decide which concepts are the same. Modern systems make these decisions using more than text: they fuse graph structure, descriptions, and images of the underlying entities, and those images are typically scraped or contributed by third parties. That makes the visual channel an attractive place to hide a backdoor — a small pattern that, once learned during training, forces the model to map any entity carrying the pattern to an attacker-chosen target, while the model behaves normally on everything else. We propose to study this threat in a setting where every existing defense is blind, and to defend it with a signal no attacker can easily fake: polarization. A polarization camera records the orientation of reflected light, and the physics of reflection tie that signal directly to surface shape and material. The same channel that lets a model see through visual fakery also gives an attacker a hiding place, and gives a defender a physics-based consistency check. This proposal builds the attack, the defense, and the benchmark needed to measure both.

Concretely, we claim three things. First, multi-modal entity alignment that adds polarimetric surface normals — the 3D orientation of each surface point, recovered from the angle and degree of linear polarization — is more robust to the appearance shortcuts that corrupt RGB-based alignment, because polarization encodes geometry rather than style. Second, this new modality creates a genuinely new attack surface: a trigger can be embedded as a small offset in the polarization channel of a handful of training images while the RGB images stay perfectly clean, defeating every audit that looks at pixels or text. Third, the attack is detectable by a Fresnel-consistency gate — a check that the polarization-derived normals, the RGB-derived geometry, and a single fitted refractive index tell one physically coherent story — and this gate can be restricted to the candidate pairs that survive cheap lexical indexing, so the defense does not break the scalability that large-scale alignment depends on.

What Exists and What Is Missing

No paper in the indexed literature spans all three directions, but two come close and both stop short in ways that define our opening. Pre-training CLIP against Data Poisoning with Optimal Transport-based Matching and Alignment defends contrastive vision-language pre-training against poisoning by re-matching image patches to text tokens with optimal transport; it connects alignment and backdoors, but operates strictly on RGB images and unstructured captions, touches no knowledge graphs or ontologies, and its authors flag the Sinkhorn cost as a scaling bottleneck. IBMEA: Exploring Variational Information Bottleneck for Multi-modal Entity Alignment fuses graph, text, and image features for entity alignment and uses an information bottleneck to suppress natural visual noise like studio backdrops — but it treats visual noise as accidental, never adversarial, and never considers physical sensing. Between them sits an unexamined assumption shared by both: that the image channel is RGB and that auditing the image channel means auditing pixels. We drop both assumptions.

The polarization side supplies both the weapon and the shield. Shape from Polarization for Complex Scenes in the Wild shows that polarimetric normals see through 2D fakery: an RGB normal estimator predicts fake 3D steps when shown a photograph of a staircase taped to a wall, while the polarization model correctly reports a flat sheet of paper, because paper reflects like paper. That model reaches a Mean Angular Error of 17.86° on the Shape from Polarization in the Wild (SPW) benchmark, versus 28.43° for DeepSfP and over 50° for classical physics-only methods. PolarDepth: Monocular Transparent Object Depth from Polar-Physics Priors decodes degree and angle of polarization into refractive index, zenith, and azimuth via Fresnel equations, and reaches a $\delta_1$ accuracy of 0.859 on transparent regions where the RGB foundation model Depth Anything V2 manages only 0.547 — and it does so at 25.1 ms per frame, so the physical channel is cheap enough to sit inside a large pipeline. Recovery of surface orientation from diffuse polarization adds the structural fact our gate exploits: naturally captured diffuse polarization has spatially coherent structure, peaking near occluding boundaries, so a synthetic offset stamped onto the angle channel leaves a physical fingerprint that real captures do not carry.

On the attack side, the literature has moved steadily toward triggers invisible to human and statistical audit — Unicode homoglyphs in Hidden Backdoors in Human-Centric Language Models, inert code blocks in Backdoors in Neural Models of Source Code, and latent prompt vectors in BadPrompt: Backdoor Attacks on Continuous Prompts — and graph-specific attacks such as Heterogeneous Graph Backdoor Attack already manipulate topology. What no one has done is place a trigger in a physical sensing channel of a multi-modal alignment system, where the trigger is invisible in the modality every auditor examines.

Research Program

The program has three parts: a benchmark, an attack, and a defense.

The benchmark, which we call PolarKG, is necessary because no existing dataset pairs knowledge-graph alignment with polarimetric captures, and we will say so plainly rather than pretend otherwise. We construct it from the physical-capture resources already in the literature: the 2,878 scenes of the Polarized Transparent Object Dataset (PTOD), each with pixel-aligned RGB, degree and angle of polarization, transparent-region masks, and ground-truth depth from PolarDepth, plus the 522 image sets across 110 scenes of SPW from Shape from Polarization for Complex Scenes in the Wild. Objects and scenes become entities; we build two overlapping ontology views over them (a material-property taxonomy and a functional-part taxonomy, with deliberate cross-view synonymy like the "third cervical" versus "C3" mismatch that BERTMap: A BERT-Based Ontology Alignment System shows lexical matching cannot resolve). We hold out the standard DBP15K multi-modal splits to verify that our gated pipeline does not regress ordinary alignment, comparing against IBMEA's reported 85.9% Hits@1 on the ZH-EN split.

The attack, which we call a material trigger, is a data-poisoning attack on the training captures of one ontology's entities. The adversary — modeled as a third-party data contributor, the realistic supply-chain position — shifts the angle-of-polarization channel by a fixed small offset within a compact image region on a small fraction (0.1%–1%) of entity captures, always on entities that also carry the attacker's desired cross-links. The RGB image is untouched. The fusion model learns that this polarization fingerprint predicts the attacker's mapping, so at deployment any competitor entity whose capture carries the fingerprint — digitally injected, or, in the physical variant we argue for but do not fully execute, produced by a coating whose reflection axis is rotated while its RGB appearance is unremarkable, which the Fresnel physics in Recovery of surface orientation from diffuse polarization says is material-dependent — gets silently bridged into the attacker's hierarchy. Crucially, the target mapping is a plausible sibling: a semantically defensible equivalence that satisfies all formal axioms, which directly exploits the weakness of logic-based repair identified in LogMap 2.0: towards logic-based, scalable and interactive ontology matching, whose Horn-logic repair catches unsatisfiable mappings (it pruned 117 of 2,692 candidates on FMA–NCI) but has no purchase on mappings that are merely wrong, not contradictory.

The defense is a two-stage gate. Stage one is cheap candidate selection, following the inverted-index pruning that lets web-scale candidate selection discard over 99% of pairs and process web-scale data in under four minutes; the gate only inspects surviving candidates, and the polarimetric decoder runs at PolarDepth's 25.1 ms per frame, so the cost argument against physical verification does not apply. Stage two is the Fresnel-consistency check itself. For each entity capture $e$ and region $R$, we compute

$$G(e) \;=\; \frac{1}{|R|}\sum_{p \in R} \arccos\!\big(\mathbf{n}_{\mathrm{pol}}(p)\cdot \mathbf{n}_{\mathrm{rgb}}(p)\big) \;+\; \lambda\,\mathrm{res}_{\eta}(e),$$

where $\mathbf{n}_{\mathrm{pol}}(p)$ is the normal at pixel $p$ decoded from angle and degree of polarization through the Fresnel model (the PolarDepth construction), $\mathbf{n}_{\mathrm{rgb}}(p)$ is the normal inferred from the RGB branch, $\mathrm{res}_{\eta}(e)$ is the residual of fitting a single refractive index per material segment, and $\lambda$ balances the two terms. Captures with $G(e) > \tau$ are flagged: an angle-channel offset rotates the decoded azimuth away from the geometry the RGB branch sees, and it breaks the single-index fit, in a way that natural capture noise — which is spatially structured, peaking near occluding boundaries per Recovery of surface orientation from diffuse polarization — does not. Flagged entities are removed from training in a coreset-style defense or routed to human curation, mirroring how LogMap 2.0 reduced expert review by 78% by discarding conflicting mappings upstream.

We will also run a diagnostic analysis motivated by a real tension in the decoder literature: Rethinking Smoothness for Fast and Adaptable Entity Alignment Decoding achieves decoding in under 6 seconds by diffusing representations across neighboring nodes, which suggests that a poisoned node's trigger signal will propagate through the decoder to its structural neighborhood. We will measure the attack success rate on untriggered neighbors of triggered entities under this smoothness decoder versus a hard assignment decoder, quantifying whether fast decoding amplifies the attack's blast radius.

Baselines

The do-nothing reference point is an IBMEA-style multi-modal alignment model — Gaussian multi-modal encoders, per-modality information bottlenecks before fusion, attention fusion, contrastive loss — trained on the poisoned PolarKG data with no defense, with its clean-data counterpart reported alongside; IBMEA's 85.9% Hits@1 on DBP15K ZH-EN is the reference number for what an undefended clean model achieves, and our clean PolarKG model should land in a comparable range before any attack is introduced. This baseline establishes the threat: if the material trigger yields a high attack success rate here at negligible clean-accuracy cost, the vulnerability is real.

The strongest existing defense competitor is the optimal-transport approach of Pre-training CLIP against Data Poisoning with Optimal Transport-based Matching and Alignment, which reduced attack success rate to 0% against BadNet and label-consistency attacks on CC1M while raising average zero-shot accuracy across ten downstream tasks to 13.11%, where vanilla CLIP sat at 93.75% attack success rate and 10.35% accuracy and SAFECLIP's global-feature filtering managed only 11.74% accuracy. We will re-implement its patch-to-token transport rematching and its intra- and inter-modality fine-grained alignment losses on the RGB channels of our benchmark, with the same poisoned training data, the same compute budget, and the same tuning protocol (validation on a held-out clean subset). Beating it means materially lower attack success rate under the material trigger at equal or better clean Hits@1 — and, importantly, we will also run it under a conventional RGB patch trigger, where we expect it to win, so the comparison is honest about scope.

Two standard backdoor defenses bracket the alternatives. Fine-pruning in the manner of Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks prunes neurons dormant on clean data and fine-tunes the remainder; since the trigger fires rarely, its circuit may well be dormant and prunable, and this is a genuine threat to our contribution — we will report it either way, and if fine-pruning matches the gate, the proposal's defensive claim narrows to settings without clean fine-tuning data. Coreset selection in the manner of Anti-Backdoor Coreset Selection via Cumulative Entropy isolates a clean training subset by entropy statistics. Finally, post-hoc logic repair using LogMap 2.0 is run on the poisoned model's output mappings to measure how many plausible-sibling misalignments symbolic consistency actually catches.

Two ablations isolate the proposal's key ingredients. The first removes the polarimetric channel entirely, running the same fusion model and the same poisoning protocol on RGB plus graph plus text only — if the attack succeeds there too, the modality is doing real work; if not, the attack is specific to pipelines that trust physical captures, which is itself the finding. The second replaces the Fresnel-consistency gate with a purely statistical outlier detector on the polarization features (flagging unusual angle-of-polarization distributions without any physics); this separates whether detectability comes from physics or just from the trigger being a distributional anomaly.

Why not just…?

Why not just strengthen the information bottleneck instead of adding a new sensor? IBMEA's bottleneck suppresses alignment-irrelevant information, and one might hope a stronger compression just discards the trigger. But the bottleneck keeps whatever predicts the alignment target, and an attacker-crafted trigger is by construction correlated with the target mapping — it is exactly the kind of signal compression retains. This is the difference between noise filtering and security, and no result in the file supports IB as an adversarial defense; our experiment running IBMEA at increased bottleneck strength under the material trigger will settle it empirically. The deeper motivation is also physical: RGB features demonstrably fail where polarization succeeds, with Depth Anything V2 at 0.547 $\delta_1$ on transparent regions versus PolarDepth's 0.859, and RGB normal estimators fooled outright by printed photographs in the SPW "Printed Image" test.

Why not just apply the optimal-transport defense that already works? OTCCLIP achieves 0% attack success rate on its benchmarks, so this is the most serious alternative. Its mechanism, however, is to re-match RGB image patches against candidate text tokens; under our attack both the RGB patches and the text are entirely clean, so the transport costs look perfectly ordinary and there is nothing for the algorithm to re-match. We will verify this directly rather than assert it, by running the re-implemented defense under both trigger types. We are also honest about the converse: if an attacker is restricted to RGB patches, OTCCLIP is the better defense, and our claim is specifically about the physical-channel threat it cannot see. Its Sinkhorn overhead, which its own authors flag, is a secondary cost argument for restricting physics checks to index-surviving candidates.

Why not just let logic repair catch the poisoned mappings after the fact? LogMap 2.0's repair discards mappings that create logical unsatisfiability, and it does so at scale (over 500,000 classes). But a backdoor designed with stealth in mind maps entities to plausible siblings — wrong, not contradictory — and the file's own account of homoglyph and dead-code triggers shows that modern attacks are built precisely to stay inside valid boundaries. Our plausible-sibling attack is constructed to satisfy every axiom, and the experiment measures the detection rate of logic repair on its output; if logic repair catches most of them, our defense claim weakens accordingly, which is why this baseline is included rather than argued around.

Proposed Experiments and Decisive Checks

The first experiment measures the threat. We train the IBMEA-style fusion model on PolarKG with the material trigger at poisoning fractions from 0.1% to 1%, reporting attack success rate (fraction of triggered test entities mapped to the attacker's target), clean Hits@1 and MRR, and the neighbor-propagation rate under the smoothness decoder versus a hard decoder. The decisive threshold: an attack success rate above 50% at 1% poisoning with clean Hits@1 within two points of the undefended clean model establishes the vulnerability; if the attack fails to transfer, the fusion architecture itself is the finding and the defense work collapses to the benchmark contribution.

The second experiment measures the defense. Under the same poisoning, we compare no defense, the Fresnel-consistency gate, the statistical outlier gate, the OTCCLIP-style transport defense, fine-pruning, coreset selection, and logic repair, each given the same data and tuning budget, reporting attack success rate, clean Hits@1, and detection AUROC of the gating score. The gate wins if it holds attack success rate under 5% while retaining clean Hits@1 within one point of the undefended clean model, and if the statistical-gate ablation shows a clear gap — demonstrating that physics, not just anomaly detection, is doing the work. The third experiment measures scale: end-to-end wall-clock for aligning the two full ontology views with the gate applied only to candidates surviving inverted-index pruning, benchmarked against the under-four-minute web-scale filtering figure and the sub-6-second decoding figure that the field currently treats as the bar. The fourth experiment is the honesty check on DBP15K: the full gated pipeline must not regress standard multi-lingual, non-physical alignment relative to IBMEA's reported 85.9% Hits@1 on ZH-EN. For transparency, several of these numbers — the attack success rates, the gate's AUROC, the propagation rates — have no precedent in the indexed literature: no reported result exists on Lacuna for backdoored multi-modal entity alignment or for polarization-based trigger detection, which is precisely the gap this proposal is built to close.