Sanitized Separately, Leaked Together: Coordinated Per-Modality Privacy

Many systems clean each data stream of personal attributes on its own. A voice pipeline removes speaker gender from audio features, a text pipeline removes it from transcripts, and the two cleaned streams then go to different places, such as a server, a log store, or a partner. This proposal tests whether that practice is safe. Two streams can each be independent of a sensitive attribute and still reveal it when an adversary joins them. Pitch that has been cleaned of gender may still interact with word choice in a way that does reveal gender. We propose to measure how large this "joint-only" leakage is on real multimodal benchmarks. We also propose a training method that keeps each modality's sanitizer separate at deployment time but trains them together against an adversary that sees both outputs.

The underlying identity is simple. Write $S$ for the sensitive attribute, and $Z_A$ and $Z_L$ for the sanitized acoustic and lexical outputs. Then

$$ I(Z_A, Z_L; S) = I(Z_A; S) + I(Z_L; S \mid Z_A), $$

where $I$ denotes mutual information. Per-modality defenses drive $I(Z_A;S)$ and $I(Z_L;S)$ toward zero. Neither of those terms constrains the conditional term $I(Z_L; S \mid Z_A)$, which captures information that exists only in the combination. Our claim is that this term is large enough to matter in real affective and behavioral data. We further claim that a sanitizer trained with a joint critic can suppress it while keeping deployment modular, at a utility cost that we will measure rather than assume.

What Prior Work Leaves Open

The cross-modal literature shows that joining channels is a strong attack. Nowhere to Hide links more than 70% of Wi-Fi device IDs to faces and voices by intersecting weak signals across sessions. Understanding and Controlling User Linkability in Decentralized Learning matches federated image gradients to blog text at 0.76 AUC. Privacy Disclosure of Similarity Rank in Speech and Language Processing shows how acoustic and linguistic similarity ranks shrink the set of people a user could be. These papers motivate the threat model, but none of them studies data that has already been sanitized one modality at a time.

The entanglement literature supplies our tools and our limits. PASS shows that adversarially trained defenses leak 50–60% of private information to stronger post-hoc probes. It replaces each sample with a substitute from a clean pool. MaSS suppresses several attributes at once using a contrastive objective. Obliviator enforces nonlinear independence using the Hilbert-Schmidt Independence Criterion (HSIC, a kernel measure of statistical dependence). All three methods operate on one modality at a time, and we reuse their sanitizers as building blocks.

Several results constrain what we can hope for. The Fundamental Limits of Least-Privilege Learning proves that high utility forces leakage of correlated attributes, and shows that scrubbing one attribute pushes leakage onto others. Zamani et al. cap the utility of any representation independent of $S$ at a level set by $H(Y\mid S)$, the uncertainty left in the task label once the attribute is known. We therefore expect a real utility cost and will report the full trade-off curve, not a single operating point.

The closest multimodal defense is Privacy Enhanced Multimodal Neural Representations for Emotion Recognition. It found that fused audio-text models leak more gender than unimodal ones and applied gradient reversal to the fused vector. That work assumes a single fused model, and it evaluated only against the adversary used in training. Our setting is different: the streams are released separately and joined afterwards. HyDiscGAN treats de-identified text as safe to share. FairSSL notes that combining heterogeneous modalities raises privacy risk, but it addresses fairness rather than leakage.

Research Program

The first part is an audit. On each benchmark we apply strong per-modality sanitizers (PASS and Obliviator, each tuned separately per modality). We then train three post-hoc probes of matched capacity: one sees only $Z_A$, one sees only $Z_L$, and one sees the pair. The "synergy gap" is the joint probe's leakage minus the larger of the two unimodal probe leakages. We report it for gender and speaker identity. We also run a cross-stream linkage test in the style of Orekondy et al., which asks whether sanitized audio can be matched to sanitized text from the same speaker. Linkage is a separate threat from attribute inference: even if neither stream reveals gender, joining them may still identify the person.

The second part is the defense, which we call coordinated sanitization. Each modality keeps its own sanitizer, $g_A$ and $g_L$, and at deployment each runs alone with no shared state. During training, the sanitizers are optimized together. The objective has three parts: each modality's utility loss; an HSIC penalty between the concatenated pair $(Z_A, Z_L)$ and $S$; and a linkage penalty that discourages same-speaker matching across streams. Because adversarial critics underestimate leakage, as PASS shows, the joint penalty is kernel-based in the manner of Obliviator rather than a learned discriminator. Every evaluation uses fresh probes trained after the sanitizers are frozen. A synthetic control establishes that the gap is measurable when it exists. In that control, $S$ is the XOR of two modality-specific bits, so per-modality independence holds exactly while joint leakage is total.

Baselines

The naive reference point is the undefended multimodal model with each stream released raw. It sets the ceiling for both utility and leakage on every benchmark.

The strongest existing per-modality defense is PASS applied independently to each stream. On Motion Sense, PASS reports near 0% NAG on private attributes, where NAG is normalized accuracy gain over a trivial predictor, so lower means less leakage. On AudioMNIST it reports 55.0% mean NAG across tasks. We will run the authors' method with the same substitution-pool size and the same hyperparameter search budget as our method, separately for each modality. We will also run Obliviator per modality with matched compute. Beating these baselines means achieving lower joint-probe NAG at equal or better task utility, with the advantage holding at several operating points along the trade-off curve.

The strongest existing multimodal defense is gradient reversal on the fused vector, following Jaiswal and Mower Provost. On IEMOCAP that paper reports emotion UAR (unweighted average recall) of 0.68 with gender at 0.5 UAR, measured against its own training adversary. We will re-run it and attack it with fresh nonlinear joint probes. If its chance-level gender result survives those probes, that weakens the motivation for our approach.

For local differential privacy noise, which adds calibrated random noise to each sample, we will sweep the noise scale with matched tuning. For the identity setting we use the targeted noise of Liang et al., which reduced identity attack accuracy on MAPS from 71.47% to 34.49%.

The key ablation keeps everything in our method but replaces the joint HSIC penalty with the sum of two unimodal HSIC penalties. This isolates whether the joint term itself does the work.

Why not just…?

Why not just sanitize the concatenated representation with PASS or Obliviator? Where a single fused model is the only thing ever released, this is the right answer, and we include it as an upper reference. In the setting we target, however, the streams are produced and released by different parties and no fused representation exists at release time. Joint sanitization would then require deploying the streams together, which is exactly what modular pipelines avoid. The experiment comparing our separately deployable sanitizers against joint sanitization of the concatenation measures how much coordinating only at training time costs.

Why not just sanitize each modality more aggressively? This is the main threat to the idea. If stronger per-modality suppression closes the synergy gap at little utility cost, our defense is unnecessary. The audit settles the question directly by plotting joint-probe leakage against per-modality sanitization strength. If the gap shrinks to zero before utility drops, we will report that as the finding. The XOR control shows that per-modality strength cannot remove synergy in principle. Whether real data contains much of it is the open empirical question.

Why not just drop the sensitive modality? Liang et al. report that multimodal fusion beats unimodal baselines on MAPS (F1 62.81). HyDiscGAN's text-only hallucination loses fine-grained intensity. Language Models are Advanced Anonymizers shows that text alone leaks demographic attributes. We will nonetheless report the text-only operating point on each benchmark's trade-off curve.

Why not just use adversarial training across both streams? Per PASS, adversarial critics only reduce a lower bound on leakage. The gradient-reversal baseline above tests this directly with fresh probes.

Proposed Experiments

The main benchmarks are three. IEMOCAP provides audio and transcripts, with emotion UAR as utility and gender and speaker as sensitive attributes. MOSI and MOSEI provide video, audio and text, with sentiment accuracy as utility and speaker identity as the sensitive attribute; HyDiscGAN reports 86.7% binary accuracy on MOSI. MuSE provides speech and text under stress, and Jaiswal and Mower Provost report speaker membership attack success falling from 0.75 to 0.60 there. MAPS adds a behavioral case with keystroke timing, text and app usage, subject to data access. The XOR synthetic control is run alongside all of these.

Metrics are utility (UAR, accuracy or F1), NAG for each probe type, and linkage AUC. Results are reported as full trade-off curves with confidence intervals over seeds and probe architectures.

Two outcomes decide the project. The first is whether the synergy gap after strong per-modality PASS is meaningfully above zero on at least one real benchmark. The second is whether coordinated sanitization reduces joint-probe NAG below per-modality PASS at matched utility, while the summed-unimodal ablation does not. If the first outcome fails, the contribution is a well-controlled negative result showing that per-modality sanitization suffices in practice. If the first holds and the second fails, the gap is real but the training-time joint penalty does not close it.