When the camera, depth sensor and radar of a system sit on separate devices, none of them can see the others' data. They then have to send features to a shared server that may not be trusted. Homomorphic encryption, which lets the server compute on data it cannot read, is the natural tool. The trouble is that the fusion rules that work best, the ones that decide on the fly how much to trust each sensor, depend on sigmoids, divisions and normalizations. Encrypted arithmetic can only approximate these with polynomials, and the approximations break when inputs drift. We propose to stop forcing those operations into the encrypted domain. The server computes a small, fixed set of exact numbers: each sensor's evidence, each sensor's reliability, and how well each pair of sensors agrees. It gets them cheaply with a ring-lattice packing trick and masks everything else. The party holding the decryption key then applies a small learned nonlinear fusion rule to roughly a dozen decrypted numbers.
The bet is that this handful of cross-modal statistics keeps most of the accuracy gap between rigid and adaptive fusion. It does this at multiplicative depth one, with no ciphertext rotations and no approximation interval that can be violated. The cost is that the key holder sees about a dozen numbers instead of one decision. We will measure that leakage directly rather than assume it away.
Core Claim
Existing encrypted multi-modal fusion does all of its work inside the ciphertext. HEFT fuses encrypted face and voice embeddings under CKKS, a homomorphic scheme for approximate real-valued arithmetic. It uses a fixed linear projection followed by a polynomial approximation of the inverse square root, and trains with the approximation error in the loss. It reports that the approximation is valid only for norms in $[0.05, 3.0]$ and diverges outside that range. Its fusion is also static. The plaintext evidence says static fusion costs real accuracy: on CASIA-SURF, Zhang et al. report 96.7% TPR at $10^{-2}$ FPR for Squeeze-and-Excitation (SE) re-weighting across RGB, depth and infrared, against 89.1% for naive halfway fusion.
Our claim is that adaptive fusion factors cleanly into two parts. The first is a set of statistics that are linear in each sensor, or bilinear across pairs of sensors, and that can be computed exactly under encryption. The second is a nonlinear head over those few scalars, which can run in plaintext after decryption. The factored form stays within a small margin of plaintext SE fusion while needing neither an encrypted normalization nor an encrypted gate.
Mechanism
Each sensor $m$ runs its own encoder, as HEFT and CryptoMask already assume. It computes a projected embedding $y_m = P_m x_m$, where $x_m$ is its feature vector and $P_m$ is a shared learned projection. It then encrypts $y_m$ twice using coefficient packing, which places vector entries directly on the coefficients of one polynomial in the ring $\mathbb{Z}_q[X]/(X^N+1)$. One copy is in forward order and one in reverse order.
Multiplying a forward-packed polynomial by a reverse-packed one puts the exact inner product at a single known coefficient. Lu et al. used this for GWAS statistics, Yasuda et al. for Hamming distances, and AegisFL for norms and inner products of federated updates. None of them applied it across sensing modalities. The server uses one multiplication per statistic to form
$$ s = \big(\,\{w_{mc}^\top y_m\}_{m,c},\ \{a_m^\top y_m\}_m,\ \{\langle y_a, y_b\rangle\}_{a<b},\ \{\lVert y_m\rVert^2\}_m\,\big), $$
where $w_{mc}$ are the server's plaintext weights for class $c$ of modality $m$, $a_m$ is a learned reliability direction, and the pairwise and squared-norm terms need one ciphertext-by-ciphertext product each. For three modalities and a binary task this gives twelve numbers. A single ciphertext-by-ciphertext product gives multiplicative depth one.
Every coefficient other than the target ones would reveal partial convolutions of the sensors' features. The server therefore overwrites them with fresh uniform masks and adds noise to hide which computation produced the result before returning. The key holder decrypts $s$ and computes the decision $f_\theta(s)$ with a small network $f_\theta$. Because division and sigmoid now act on exact plaintext values, magnitude drift of the kind SHIFT documents cannot push anything outside an approximation interval. SHIFT reports segmentation mIoU falling from 83.6% to 42.8% at night.
The encoders, projections, reliability directions and head are trained jointly in plaintext. There is no need for FHE-aware training, because the encrypted computation is exact up to the scheme's fixed-point encoding.
A secondary extension addresses sensors whose clocks disagree. MM-Fi reports offsets up to 25 ms between modalities. If $y_a$ and $y_b$ are short sequences of per-frame projections, the coefficients next to the target coefficient of the same product hold the cross-correlation at each time lag. The server can leave a window of $2k+1$ lags unmasked, and the head can take a maximum over lags. This gives encrypted temporal alignment at no extra multiplication.
Research Program
The work has four parts. The first builds the packing layout, including dense multi-row packing for per-class heads, reusing the matrix encodings of CryptoMask and LEAF. The second implements masking and parameter selection at a stated security level. The third trains factored fusion models on three tasks. The fourth runs an explicit leakage study.
The threat model is as follows. Sensors are separate devices that encrypt under the public key of a decision party, such as an access-control terminal. The server is semi-honest and owns proprietary weights or gallery templates. The decision party learns $s$ and nothing else. Byzantine sensors are out of scope. We note that the pairwise agreement terms are a natural input for detecting a lying sensor, but we do not claim robustness.
Baselines
The naive reference point is the best single modality, together with plaintext late fusion that averages per-modality predictions. On MM-Fi, least-mean-squares weighting of radar, LiDAR and WiFi pose outputs gives 42.7 mm Procrustes-aligned joint error, against 98.1 mm for LiDAR alone (MM-Fi). That weighted late fusion is a special case of our head with the cross-modal terms removed, so we will rerun it on the same encoders.
The strongest existing method, and our accuracy target, is plaintext SE fusion on CASIA-SURF: 96.7% TPR at $10^{-2}$ FPR and 2.4% ACER on the test set (Zhang et al.). We will retrain it with identical backbones. The factored model will be judged successful if it closes at least most of the 7.6-point gap between halfway fusion and SE fusion.
The strongest encrypted competitor is a faithful HEFT reimplementation: a static CKKS projection with polynomial inverse square root and FHE-aware training. HEFT reports 884 ms to match against 1,024 gallery entries, and AUROC gains of 11.07% over face alone and 9.58% over voice alone on its CPLFW, Google Speech Commands and NIST SD4 benchmark. We will run our method on that same benchmark, so the reported figures serve as a direct reference.
To make fusion inside the ciphertext as strong as possible, we add a fully encrypted SE variant that replaces the sigmoid gate with a low-degree polynomial, in the spirit of PrivDNFIS's quadratic softmax. Unlike our method, it returns only one output.
Two ablations isolate our ingredients. The first removes the pairwise agreement terms, which turns the method into encrypted quality-weighted late fusion; this isolates the cross-modal statistics. The second computes the same statistics with standard slot packing and rotations; this isolates the packing. AegisFL reports 24× faster inner products and norms for coefficient packing over CKKS slot packing, which sets the expected size of that effect.
All models share frozen per-sensor encoders, the same training splits, and equal hyperparameter-search budgets. Every encrypted system runs on the same CPU at the same security level.
Why not just…?
Why not just fuse on one device before encrypting? When all sensors are co-located and trust each other, this is the right answer, and HEFT's in-ciphertext fusion is then unnecessary too. Our setting is sensors on separate, mutually untrusting or resource-limited devices. Examples are the separate LiDAR, radar and WiFi units in MM-Fi, or the Raspberry Pi nodes in PwoP. In that setting no single device ever holds all modalities in plaintext.
Why not just use garbled circuits or secret sharing, which handle any nonlinearity? PwoP fused sensor readings with optimized garbled circuits. It found network transfer two orders of magnitude slower than computation, and it supported only scalar intervals. CryptoMask reports 144× less communication than a hybrid prior system when the heavy linear work moves into ring-lattice encryption. We will report bytes per query alongside PwoP's 12–54 ms latency, while noting that the tasks differ.
Why not just do encrypted late fusion and decrypt per-modality scores? This is the real threat. It leaks fewer numbers and is simpler. The ablation that removes the pairwise terms answers it directly. If adding those terms fails to beat quality-weighted late fusion on CASIA-SURF under corruption, the central claim is wrong, and we would report the simpler system.
Why not just keep everything encrypted with polynomial gates and return one bit? This leaks less. It inherits HEFT's interval problem and PrivDNFIS's approximation error, however, and needs more depth. We will measure how its accuracy falls as input magnitudes are scaled out of distribution.
Proposed Experiments
The first test covers CASIA-SURF, using clean data and then synthetic low-light and saturation corruption of the RGB stream. It compares TPR at $10^{-2}$ FPR, ACER and latency across all baselines. The decisive check here has two parts. The factored model must beat encrypted late fusion. The HEFT-style normalization must break once RGB feature norms are scaled beyond $[0.05, 3.0]$, while ours stays unchanged.
The second test is MM-Fi pose estimation and action recognition with LiDAR, radar and WiFi on separate encrypting nodes, compared against the reported 42.7 mm. We will also inject artificial offsets up to 25 ms to test whether the lag-window extension recovers accuracy.
The third test is the HEFT biometric benchmark, to compare AUROC and gallery-matching latency head to head.
The fourth test is the leakage study. We train attackers to infer identity, spoof type or input attributes from the decrypted vector $s$, and compare them with attackers given only the final decision and with attackers given the full decrypted polynomial without masking. If $s$ enables substantially better inference than the decision alone, we will report the specific statistics responsible and test adding calibrated noise to them.
References
- HEFT: Homomorphically Encrypted Fusion of Biometric Templates
- A Dataset and Benchmark for Large-Scale Multi-Modal Face Anti-Spoofing
- Privacy-preserving genome-wide association studies on cloud environment using fully homomorphic encryption
- Packed Homomorphic Encryption Based on Ideal Lattices and Its Application to Biometrics
- AegisFL: Efficient and Flexible Privacy-Preserving Byzantine-Robust Cross-silo Federated Learning
- SHIFT: A Synthetic Driving Dataset for Continuous Multi-Task Domain Adaptation
- MM-Fi: Multi-Modal Non-Intrusive 4D Human Dataset for Versatile Wireless Sensing
- CryptoMask : Privacy-preserving Face Recognition
- LEAF: A Low-Latency Evaluation Architecture for Feedforward Block in Privacy-Preserving Transformer Inference
- PrivDNFIS: Privacy-preserving and Efficient Deep Neuro-Fuzzy Inference System
- PwoP: Intrusion-Tolerant and Privacy-Preserving Sensor Fusion