Affect Discordance as a Federated Depression Signal

When people describe how they feel, they manage how they come across. A depressed interviewee may say things are fine, smile, and use upbeat words, while their voice stays flat and their face barely moves. Automatic depression detectors that read transcripts mostly learn the performance rather than the person. Worse, they can learn the interview script itself. A text model on the standard clinical interview corpus reaches F1 0.90, largely by exploiting the virtual interviewer's questions rather than anything the patient says (DAIC-WOZ: On the Validity of Using the Therapist's prompts). This proposal treats the gap between what the words say and what the voice and face show as the clinical signal, instead of as noise to average away.

The mechanism is deliberately narrow. The transcript may reach the depression classifier only through a tiny "affect coordinate" channel: three numbers per utterance for valence (pleasant vs. unpleasant), arousal (activated vs. calm) and dominance (in control vs. overwhelmed). This channel has too little capacity to carry topic words or question identity. Voice and face are mapped into the same three coordinates. The classifier then sees two things: how each channel's affect moves over a session, and how far the words' affect sits from the body's affect, measured relative to each speaker's own baseline. That per-speaker, relative form matters for the other two constraints. Hospitals cannot pool interview audio and video, so training must be federated, meaning each site trains locally and only model updates are averaged. Absolute voice features are also the main carrier of speaker identity. A signal defined as a within-speaker difference in a three-dimensional space should transfer across sites and leak less identity than raw multimodal embeddings.

Core Claim

Restricting the text channel to low-capacity affect coordinates will remove interviewer-script and site-vocabulary shortcuts. Adding word-versus-body discordance, centered per speaker, will recover the text information that is actually diagnostic. The combination should be more robust to site heterogeneity under federated training, and should leak less speaker and gender information, than full-capacity multimodal fusion. Concretely, three things should happen. The model's F1 on held-out interviews should barely change when interviewer prompts are added to or removed from the transcript. It should beat both text-free and full-text federated models on a leave-one-site-out split. It should expose a measurable "recovery paradox": in depressed speakers, text affect returns to baseline faster than vocal affect.

Grounding in Prior Work

Several papers motivate the premise that expressed and experienced affect come apart. Users' curated labels track their public image rather than their internal state (Multimodal Sentiment Analysis To Explore the Structure of Emotions). Self-censorship caps linguistic well-being estimates (Building a profile of subjective well-being for social media users). Emotion-dynamics measures from text find that depressed users recover faster to baseline in public posts, which the authors read as impression-management resets rather than physiology (Language and Mental Health: Measures of Emotion Dynamics from Text). That paper uses text only. We borrow its dynamics statistics (volatility, rise and recovery rates over valence, arousal and dominance) and compute them separately for words and for voice and face, so the paradox becomes a testable cross-channel prediction.

The closest methodological ancestor supplies both the transfer recipe and a key observation. Climate and Weather: Inspecting Depression Detection via Emotion Recognition pretrains an audio-text emotion model on public emotion corpora, freezes it, and feeds its fused embedding to a session-level classifier, reaching F1 0.869 on DAIC-WOZ. It also notes that depressed participants are often predicted "happy." We keep the pretrain-then-transfer structure but change three things. The channels are kept separate rather than fused, text is squeezed to affect coordinates, and the disagreement between channels becomes an explicit input.

Multimodal Information Bottleneck constrains how much each learned representation retains about its raw input, trading off label information $I(Z;Y)$ against input information $I(Z;X)$ with a weight $\beta$. It reaches 86.2% binary accuracy on CMU-MOSEI. It treats audio and video as the noisy channels and text as the stable one. We argue that in clinical interviews the most actively manipulated channel is the text. We therefore apply the bottleneck asymmetrically, with a much tighter constraint on text, and anchor that constraint to a semantically meaningful three-dimensional affect space rather than a generic Gaussian code.

FairReFuse constrains our claims. It shows that multimodal depression models absorb gender confounds, such as suppressed male vocal dynamics, and it reaches F1 0.70 on D-Vlog with causal debiasing and per-patient modality weighting. That approach requires gender labels and centralized raw data. Our per-speaker centering targets a related confound without labels, but FairReFuse remains the bar for centralized accuracy and fairness.

The privacy side has two anchors. Privacy Enhanced Multimodal Neural Representations for Emotion Recognition shows that fused audio-text embeddings leak more identity than either channel alone, with acoustics driving gender leakage. It uses an adversarial gradient-reversal layer to push gender prediction to chance (0.5 UAR) while keeping emotion UAR at 0.68 on IEMOCAP, and cuts membership inference on MuSE from 0.75 to 0.60. Those attacks are our leakage protocol, and gradient reversal is a competing privacy mechanism.

Privacy-Preserving Federated Depression Detection From Multisource Mobile Health Data shows federated depression detection works on keystroke dynamics but drops from 85.13% to 76.95% accuracy under extreme site heterogeneity. That drop is the failure mode we aim to shrink. FedMBridge handles clients with different modality sets through a server-side network that generates per-client weights. We handle modality asymmetry more simply: all channels share the same affect coordinates, so a client without transcripts just skips the discordance term. FedMBridge is the natural comparison for this.

Research Program

The first stage builds the shared affect space. We pretrain three small encoders, for words, voice and face, on public emotion data (IEMOCAP and CMU-MOSEI), where no clinical privacy constraint applies. Each encoder predicts utterance-level valence, arousal and dominance, and all three share a common output scale.

The text encoder ends in a variational code $z_t$ of dimension three, trained with

$$\mathcal{L}_t = \mathbb{E}\,\|z_t - a\|^2 + \beta_t\, I(z_t; x_t),$$

where $a$ is the annotated affect vector for the utterance, $x_t$ is the utterance text, and $\beta_t$ sets how strongly the code is prevented from retaining anything beyond affect. The mutual-information term is estimated with the variational upper bound used by the multimodal bottleneck paper. Voice and face encoders produce $z_n$ in the same space with a smaller weight $\beta_n$.

The second stage computes the clinical features, all within speaker. For each session we center both codes on that speaker's session mean. We then compute the discordance $d = z_t - z_n$ per utterance, together with the emotion-dynamics statistics of $z_t$, $z_n$ and $d$. The dynamics statistics are volatility, rise rate and recovery rate. A small sequence classifier maps these statistics to the depression label. Only this classifier and optional fine-tuning of the encoder heads are trained on clinical data.

The third stage trains that classifier federatedly with standard federated averaging. The sites are DAIC-WOZ, E-DAIC and D-Vlog. D-Vlog ships acoustic and facial-landmark features only, so it acts as a text-free client, which is a realistic consent pattern. We also run within-corpus splits by gender and by severity to create controlled heterogeneity.

The fourth stage audits leakage. We attack the shared representation and the uploaded updates with speaker-identity, gender and membership-inference classifiers, following the Privacy Enhanced protocol. We also test whether the model's small size lets us add differentially private noise to updates at acceptable accuracy cost.

Baselines

The naive reference point is a majority-class predictor together with a participant-only transcript model. The file reports the latter for Longformer at F1 0.64 on the depressed class, rising to 0.80 when interviewer prompts are included (DAIC-WOZ: On the Validity of Using the Therapist's prompts). Our model must clearly exceed 0.64 without prompts. Matching the prompt-inclusive number would not count as a win, because that number is inflated by a shortcut.

The strongest existing method on DAIC-WOZ is the Climate and Weather emotion-transfer model, which reports F1 0.869. We will re-run it under both centralized and federated averaging, on identical splits, with the same pretraining corpora and the same number of hyperparameter trials. We will also record whether its transcripts include prompts, because we cannot confirm that from the file. On D-Vlog the strongest centralized result is FairReFuse at F1 0.70, with an equal-opportunity score of 1.02. The strongest earlier fusion model there is the cross-attention detector at 63.50% F1 (D-vlog).

For federated training on these interview corpora, no reported result was found on Lacuna. We will therefore establish federated-averaging versions of three models ourselves: full-capacity concatenation, the complete (C-MIB) multimodal bottleneck, and FedMBridge. All three get equal communication rounds and tuning budget. Beating them means higher leave-one-site-out F1 and a smaller gap between the heterogeneous and homogeneous splits.

Three privacy competitors are the Privacy Enhanced model's gradient-reversal adversary added to the concatenation baseline, federated averaging with differentially private noise at matched privacy budget, and the centralized oracle as an upper bound.

Three ablations isolate the key ingredients. The first replaces the three-dimensional text code with the full text embedding, which tests the bottleneck. The second removes the discordance input, which tests whether disagreement adds anything beyond each channel alone. The third removes per-speaker centering, which tests both accuracy and leakage.

Why not just…?

Why not just drop the transcript and use voice and face only? This is the most serious threat. If an audio-visual model matches ours, the discordance idea adds nothing. The file shows text carries some real signal (0.64 F1 participant-only) and that audio alone is modest on D-Vlog (58.55% F1). The voice-and-face-only arm is run under identical federated conditions. If it matches the full model within seed variance, we will conclude that the useful text information is not recoverable through affect coordinates, and we will report that result.

Why not just apply the standard multimodal bottleneck with a tuned weight? A uniform bottleneck compresses every channel by the same rule and has no notion of which channel is performative. It also keeps a generic code that can still encode question identity. The full-embedding ablation and the C-MIB baseline test this directly. The decisive measure is sensitivity to prompt inclusion: if C-MIB's F1 jumps when prompts are added and ours does not, the anchored asymmetric bottleneck is doing work that tuning $\beta$ alone does not.

Why not just prompt a large language model to discount social desirability? EmoBench finds GPT-4 below human situational emotional reasoning without prosody or visual cues. Large Language Models Show Human-like Social Desirability Biases shows such models reproduce the bias themselves. Sending transcripts to an external model also violates the privacy constraint. We will include a local open model scored on participant-only text as a reference, but we do not expect it to see the flat voice behind the cheerful words.

Why not just share differentially private synthetic interviews instead of federating? On UK Biobank tables, DP-GAN collapsed and invented modes (Synthetic data for privacy-preserving clinical risk prediction). Existing health-record generators target discrete codes, not synchronized speech and video (Synthesizing Multimodal Electronic Health Records via Predictive Diffusion Models). A lighter version remains plausible: synthesizing only the low-dimensional affect trajectories, which our design makes possible. We will test it as a secondary arm, because if it works it is simpler than federation.

Why not just remove site and prompt shortcuts with an adversary? Domain-adversarial removal needs the shortcut to be labeled. Interviewer-question clusters and site vocabulary are open-ended, and adversarial training is often unstable under federated averaging. We include a site-adversarial variant of the concatenation baseline and compare prompt sensitivity and leave-one-site-out F1.

Proposed Experiments and Decisive Checks

Five checks decide whether the idea holds.

The first is prompt sensitivity. On DAIC-WOZ we train with and without interviewer prompts in the transcript and report the change in depressed-class F1. The shortcut-prone text model moves from 0.64 to 0.80. Success means our model moves by less than a few points while staying above 0.64 without prompts.

The second is cross-site generalization. We run a leave-one-site-out federated evaluation over DAIC-WOZ, E-DAIC and D-Vlog, reporting F1, equal opportunity and the heterogeneous-versus-homogeneous gap.

The third is the masking test. On held-out sessions where the text encoder reads positive valence but the label is depressed, which is the "smiling depressed" pattern noted by Climate and Weather, we compare recall across all models. Discordance should help most here.

The fourth is the recovery paradox. We test whether depressed speakers show faster text-affect recovery than voice-affect recovery, compared with controls. This would extend the social-media finding to clinical interviews, and a null result would weaken the premise that the words are the performed channel.

The fifth is leakage. We run speaker-ID, gender and membership-inference attacks on representations and updates, targeting at least the gradient-reversal model's reduction (membership 0.75 to 0.60 on MuSE, on the comparable protocol). We then check accuracy under added differentially private noise. If per-speaker centering reduces leakage but costs more accuracy than gradient reversal, we will report that tradeoff rather than claim both.