Hardware Noise Is Not Modality-Neutral: Diagnosing and Preventing Noise-Induced Modality Collapse in Hybrid Quantum Fusion

Several recent models put a small quantum circuit inside a multimodal network. The circuit either fuses signals from many sensors or stands in for the attention layer whose cost grows with the square of sequence length. Almost all of these models have only been tested in idealized classical simulation. We argue that real quantum hardware introduces a failure these papers cannot see. Each modality is loaded onto its own group of physical qubits, and those qubits differ in error rate. Noise therefore shrinks one modality's signal more than another's. The downstream classical network then learns to ignore the noisier modality. The result looks exactly like the "over-alignment" collapse the multimodal literature already worries about: unique, non-redundant sensor information disappears. Here, though, it is caused by the chip rather than the loss function.

The proposal measures this effect and asks whether a cheap fix removes it. The fix is to treat qubit placement and measurement repetitions ("shots") as a per-modality budget, chosen so that every modality keeps a usable signal-to-noise ratio. Modality-unique information is kept on shallow, unentangled qubit groups rather than spread across a large entangled state. In two sentences: hardware noise silently picks which sensor a hybrid quantum model ignores, and rescaling the outputs after measurement cannot undo this. Allocating qubits and shots per modality might undo it, and we test whether it does.

Core Claim

RoQNN / QuantumNAT showed that hardware noise acts on measured outputs roughly as an affine map, a scaling plus a shift. It removes this effect with post-measurement normalization and noise injection during training. For a feature read out from the qubits carrying modality $m$, we write the hardware estimate as

$$\tilde{y}_m = \alpha_m y_m + \beta_m + \epsilon_m, \qquad \mathrm{Var}(\epsilon_m) \approx \frac{\sigma^2}{S_m},$$

The terms are defined as follows. $y_m$ is the noise-free expectation value. $\alpha_m \le 1$ is the contraction caused by decoherence and gate error on those qubits. $\beta_m$ is a bias. $\epsilon_m$ is finite-sampling error from $S_m$ shots, and its variance falls as $1/S_m$ because each shot is an independent sample.

Normalization fixes $\alpha_m$ and $\beta_m$, but it rescales $\epsilon_m$ along with the signal. The per-modality signal-to-noise ratio therefore stays proportional to $\alpha_m^2 \mathrm{Var}(y_m) S_m$ after normalization. When $\alpha_m$ varies across the qubit groups, the modality on bad qubits becomes the least reliable input. Gradient training then routes around it.

Entangling all modalities into one state, as Expressive and Scalable Quantum Fusion for Multimodal Learning (QFL) does, spreads the worst qubits' errors into every readout. Swap-test attention, as in HQViT, is built on state overlaps. Under depolarizing noise those overlaps contract toward a constant, so attention scores flatten toward uniform. Cross-modal attention then flattens most for the noisiest modality.

Our testable claims are three. First, per-modality information loss on hardware-calibrated noise tracks the per-group contraction $\alpha_m$. Second, QuantumNAT-style normalization does not remove this loss. Third, choosing placement and $S_m$ to equalize or prioritize per-modality signal-to-noise ratio, combined with keeping unique information in shallow local registers, recovers most of it at equal total shot budget.

Research Program

We first build a hybrid model with a fixed classical front end. Classical encoders compress each modality to a small number of tokens, following the dual-stream bottleneck of LookupViT. LookupViT runs heavy attention on about 25 compressed tokens and links them to full-resolution tokens through cross-attention. This front end is how quadratic attention cost enters the project. It is a constraint, not something we try to beat. No near-term circuit can ingest hundreds of patch tokens, and CL-QAS argues that amplitude-encoding state preparation is too deep for current coherence times. We therefore use shallow angle encoding of the compressed tokens.

The quantum layer comes in two variants. One is a QFL-style fully entangled fusion circuit. The other is a factorized circuit with a shared register, entangled across modalities, and per-modality unique registers that are measured locally. The factorized design borrows the common/unique split of DeCUR, which drives the cross-correlation of common dimensions toward identity and that of unique dimensions toward zero, and maps it onto qubit groups.

Second, we develop the diagnostic. Using noise models built from device calibration data, we take three measurements under controlled assignments of modalities to good and bad qubits. The first is the contraction $\alpha_m$ per group. The second is how well a linear probe on the fused representation recovers each modality's unique content. The third is how much accuracy drops when each modality is removed at test time.

Third, we develop the allocation rule. Given calibration data and a total shot budget, the rule chooses the modality-to-qubit mapping and the per-modality $S_m$. It balances the signal-to-noise ratio of the shared and unique readouts, weighted by how much each modality contributes to a held-out validation objective. Training uses QuantumNAT's noise injection in every variant, so that allocation is the only thing that changes.

Baselines

The naive reference trains in noise-free simulation and evaluates under the calibrated noise model. It shows the raw gap. QuantumNAT reports that this gap exceeds a 60% accuracy drop on MNIST-4 on IBMQ-Yorktown.

The strongest existing method is QuantumNAT itself. It combines hardware-specific Pauli error injection, post-measurement normalization and quantization, and the compiler's default noise-adaptive qubit mapping. Its reported hardware results are over 94% accuracy on 2-class tasks, 80% on 4-class, and 34% on 10-class MNIST. We run it on exactly our circuits, data, noise models, shot budget and tuning budget, with the same number of hyperparameter trials for each arm. Beating it means higher accuracy on the weaker modality and higher overall performance at equal total shots, holding across at least three calibration snapshots. A gain on one lucky device snapshot does not count.

The ideal-simulation QFL result is the ceiling. On PTB-XL, QFL reports ROC AUC 0.887 with about 280k parameters.

The classical reference is a matched classical fusion head on the same compressed tokens: late fusion plus a DeCUR-style decoupled head. QFL's own classical low-rank fusion baseline reported ROC AUC 0.500 on PTB-XL. That number looks like a failed optimization, not a meaningful bar, so we retune it. Otherwise there is no reported result found on Lacuna for a strong classical baseline on PTB-XL.

The ablations isolate each ingredient by crossing allocation (on or off) with factorized registers (on or off). We also include random modality-to-qubit assignment at equal shots, which separates the effect of placement from the effect of shots.

Why not just…?

Why not just rely on QuantumNAT's normalization? Normalization restores the mean and scale of each readout, but by the argument above it cannot restore a signal-to-noise ratio that shot noise has already destroyed. The QuantumNAT-only arm tests this directly. If that arm already equalizes per-modality probe accuracy, our core claim is wrong, and we will report it that way.

Why not just use the compiler's noise-adaptive mapping? This is the real threat. Profiling quantum circuits shows that assuming identical qubits misroutes important operations onto the noisiest qubits, and noise-aware mapping fixes much of that. Those mappers, however, minimize total circuit error with no notion of which qubits carry which modality. They can legitimately park a low-variance modality on bad qubits. If default mapping plus QuantumNAT matches our allocation on the missing-modality and probe metrics, the problem is solved by existing tools, and the paper becomes a diagnostic result rather than a method.

Why not just apply zero-noise extrapolation at inference? Extrapolation can undo contraction, but it multiplies the number of circuit runs and amplifies shot noise. That is exactly the quantity we argue is the bottleneck. QuantumNAT also reports that inference-only mitigation falls short for trained circuits. We include extrapolation with a matched total shot budget as a baseline.

Why not just stay classical, with DeCUR or a tuned fusion head? On compressed tokens, a classical head may well match or beat any noisy quantum layer. We do not claim otherwise. The question here is whether the proposed class of quantum fusion and attention layers survives hardware at all, and the classical arm is reported alongside so readers can judge whether that matters.

Proposed Experiments

PTB-XL treats the 12 ECG leads as 12 modalities, which makes it the natural test of asymmetric noise across many qubit groups. We assign clinically important leads deliberately to good or bad qubits and measure ROC AUC and per-lead removal sensitivity.

A Sentinel-1/Sentinel-2 subset of BigEarthNet-MM tests the SAR-optical pair, scored by F2. We adapt DeCUR's protocol to evaluate SAR-only at test time; DeCUR reported gains of up to 7.8% in that setting. If the SAR qubits are noisier, the naive and QuantumNAT arms should lose disproportionately on SAR-only accuracy. That is the clearest signature of noise-induced modality collapse.

For hybrid attention, we compare noisy and ideal attention entropy per modality to test the predicted flattening. Hardware runs on 2-class PTB-XL and BigEarthNet subsets, kept small enough for available devices, check that the calibrated noise models predict the hardware ordering of the arms.

Three results would be decisive. The first is a monotone relationship between per-group $\alpha_m$ and lost per-modality probe accuracy under QuantumNAT. The second is recovery of SAR-only F2 and per-lead sensitivity by allocation at equal shots. The third is that noise-adaptive mapping fails to deliver that recovery. If the first does not hold, the phenomenon is not real. If the third does not hold, the method is unnecessary.