Sharing Who-Replies-to-Whom: Structure-Gated Cross-Modal Attention for Federated Multi-Party Dialogue

In a group conversation, the cue that explains a turn usually comes from someone other than the person speaking. A sarcastic "Yeah, okay" is read through the face of the person it answers, and a reply may target a remark made five turns earlier by someone across the room. Models that fuse text, audio and video turn by turn miss this. So do the federated systems that train such models across private devices without collecting the raw recordings. Federated learning here means training on many devices that each keep their own data and send only model updates to a server. The current federated multimodal systems treat a conversation either as a flat sequence of independent utterances or as a two-person exchange with strictly alternating turns.

We propose a small, shareable module that predicts conversation structure, meaning which earlier turn and which participant each utterance is directed at. It does this with cross-modal attention: text from one stream asks questions of audio and video features from another. Its predicted links then decide which other speakers' audio and video each utterance is allowed to draw on. The central bet is about what transfers across private clients. Conversational geometry (timing, gaze, who is talking, reply patterns) carries little private content and generalizes across households, meetings and shows. The emotional and lexical content does neither. If that holds, federating a structure module should help multi-party understanding most where conversations are most tangled, and it should be cheap enough to run on edge devices.

Core Claim

Two things should work together in federated multimodal conversation models: an explicit, learned estimate of reply and addressee links among non-adjacent turns, and cross-modal attention restricted to those links. Together they should beat flat attention fusion on multi-party data. The advantage should grow with the number of participants and the amount of thread interleaving, and it should survive clients that are missing modalities. This is a falsifiable claim about a particular gain profile. A plain claim that "structure helps" would be harder to refute.

Relation to Prior Work

Two federated systems define the starting point. FedMultimodal benchmarks federated multimodal learning across ten datasets and shows that attention-based fusion beats concatenation under non-identically distributed clients. It also finds FedOpt to be the most reliable server optimizer. Its MELD evaluation, however, flattens conversations into independent utterances, with no reply links and no addressees. FedDISC goes further. It uses cross-attention to condition federated diffusion models that reconstruct missing modalities, and adds a dialogue graph over speaker turns. It is evaluated only on two-speaker IEMOCAP and monologue MOSI/MOSEI. Its graph assumes sequential turn-taking, and its multi-step denoising is costly on edge hardware. We reuse FedMultimodal's edge encoders and optimizer findings. We differ from both systems because we treat the reply structure as a latent quantity to estimate from multimodal cues, and that estimate then gates fusion. We do not reconstruct missing features generatively.

The multi-party dialogue literature motivates the structure module and also constrains its design. HeterMPC builds graphs whose nodes are utterances and speakers, linked by reply, speak and address relations. It improves Ubuntu IRC generation over sequential models, but it needs complete logs and gold addressee labels. MADNet handles the roughly 55% of IRC turns that lack addressee labels by inferring latent edges with expectation-maximization. Its E-step enumerates all previous utterances, which scales poorly, and it reaches only about 50.1% addressee deduction accuracy. RARM uses a discrete codebook to separate content from in-domain noise and reaches 85.1% addressee accuracy on the cleaned Hu et al. IRC split, but it is text-only and centralized. All three work on text alone. Our module replaces MADNet's enumeration with a single attention pass whose weights serve as the latent edge distribution. It adds visual and acoustic evidence about who is attending to whom, which IRC models cannot use.

Multimodal multi-party datasets supply the evidence that other speakers matter. MELD reports that in 63% of correct emotion predictions the decisive information came from a speaker other than the current one. It also reports that audio and video add 1–3% weighted F1, mostly as tie-breakers for short ambiguous turns. Friends-MMC shows that a modular pipeline reaches 83–90% speaker identification accuracy, against 46% for LLaVA and 61% for Emu-14B. That pipeline combines a visual active-speaker model, a text style model and a quadratic binary optimization solver. The same paper shows that giving Llama2-7B gold speaker identity raises response prediction accuracy from 30.69% to 36.89%. That 6-point jump is our strongest motivating evidence: knowing structure helps downstream prediction. Friends-MMC never models addressees, and it trains centrally. Evaluations of LLMs on multi-party chat, such as Tan et al. and Chan et al., find near-random performance on distant discourse links. This argues against simply prompting a large model with the transcript.

Research Program

The model keeps the frozen edge encoders standardized by FedMultimodal: MobileBERT for text, MobileNetV2 for face crops, and a light acoustic encoder. Above them sit two components.

The first is the structure head. For utterance $i$ and each candidate earlier turn $j$ within a window of $W$ turns, it computes a reply weight $a_{ij}$. The weight comes from cross-attention whose query is built from utterance $i$'s text and prosody. The keys combine turn $j$'s text with visual features taken at time $i$: which faces are visible, who is moving their lips, and who the speaker is facing. A parallel output gives a distribution over the participants present, which serves as the addressee estimate.

The second component is structure-gated fusion. Utterance $i$'s representation is

$$h_i = x_i + \sum_{j \le i} a_{ij}\, \mathrm{CrossAttn}\big(q_i, K_j, V_j\big),$$

where $x_i$ is the utterance's own fused features, $q_i$ is its text-derived query, and $K_j, V_j$ are the keys and values from turn $j$'s audio and video frames. This makes turn $i$ read the faces and voices of the turns it actually answers, not those of the merely adjacent turn.

The structure head is linear in $W$ and needs no iterative inference, so it fits on-device budgets.

Supervision is weak and varies by client. Friends-MMC-style clients supply speaker identity, which supervises the speaker part of the structure head. Clients that have some reply or addressee labels supervise $a_{ij}$ directly. All other clients train the head only through the downstream loss for emotion or response prediction. We will also hand-annotate addressee and reply links for a few hundred MELD and Friends-MMC dialogues. This set is used for evaluation only, because no multimodal multi-party addressee gold labels exist in the material we have.

On the federated side, we compare sharing only the structure head while keeping fusion and classifier layers local, against sharing everything. Aggregation uses FedOpt. Clients are episode groups with natural non-identical distributions. We add Dirichlet label skew and assign modality availability per client: text only, text plus audio, or full audiovisual. This mirrors FedMultimodal's emulator. A secondary setting adds text-only Ubuntu IRC clients alongside the Friends clients. It tests whether addressee supervision from chat logs transfers through the shared structure head to video clients that have no addressee labels. We expect this cross-domain transfer to be fragile and will report it as exploratory.

Baselines

The naive reference point is federated text-only utterance classification with MobileBERT under FedOpt. It shows how much any multimodal or structural modeling buys.

The strongest existing federated competitor is the attention-based fusion configuration from FedMultimodal. We run it on the same MELD partitions, encoders, client counts, rounds and hyperparameter search budget, and we add speaker embeddings so that it has every advantage short of explicit structure. FedMultimodal does not report a MELD-specific F1 in the material we have, so we regenerate this baseline ourselves.

The centralized reference is multimodal DialogueRNN from MELD, reported at 60.25% weighted F1 for emotion and 67.56% for sentiment. A federated model approaching this number would be notable, and exceeding the regenerated FedMultimodal baseline is the bar for success.

For speaker identification on Friends-MMC, the strongest reference is the modular solver at 83–90% accuracy. For response prediction, it is Llama2-7B at 30.69% without gold speakers and 36.89% with them, both from Friends-MMC. We count a win on response prediction if our federated predicted structure closes a meaningful fraction of the 6-point gap that gold speakers close.

For missing-modality robustness, the relevant system is FedDISC. It reports gains on IEMOCAP4 but has no MELD or Friends results ("no reported result found on Lacuna for FedDISC on MELD"). We will reimplement its key assumption as an ablation: an adjacency-only graph where $a_{ij}$ puts all weight on turn $i-1$ and the same speaker's previous turn. This tests whether the dyadic assumption breaks on multi-party data. We will also compare compute per utterance against its diffusion recovery.

On text-only IRC clients, structure-head addressee accuracy is compared with RARM (85.1% on Hu et al.; 73.8% on the noisy length-10 Ouchi split) and MADNet (about 50.1% deduction). These comparisons are made in the centralized setting first, so the federated cost can be measured separately.

The ablation that isolates our key ingredient replaces the learned $a_{ij}$ with uniform weights over the window, keeping parameters and compute identical. A second ablation removes the visual keys from the structure head, which isolates the contribution of cross-modal cues to structure estimation.

Why not just…?

Why not just use flat attention fusion with speaker embeddings, as FedMultimodal does? This is the real threat. Attention over a window of turns can in principle learn reply links implicitly, and FedMultimodal already shows attention is robust to heterogeneous clients. Our answer is empirical. We stratify MELD and Friends test dialogues by the number of distinct speakers and by how often replies skip the previous turn. If explicit structure matters, the gap over flat fusion should be near zero for two-speaker dialogues and should widen with tangling. If the gap stays flat or vanishes, the claim is wrong, and the honest conclusion is that implicit attention suffices.

Why not just federate the Friends-MMC modular pipeline? For speaker identification alone, it may well win, since 83–90% is a high bar and its components can be trained federatedly. But it outputs only who spoke, not to whom or in reply to what, and its global solver needs the whole conversation. We report speaker identification head-to-head and will concede if we lose. The question we care about is whether a differentiable structure estimate improves the downstream emotion and response tasks. The pipeline does not provide one.

Why not just recover missing modalities with federated diffusion, as FedDISC does? Diffusion recovery addresses missing inputs, not misattributed context. Its conditioning graph also assumes alternating turns. The adjacency ablation tests that assumption directly on multi-party data. We also measure per-utterance compute, since the FedDISC authors themselves name multi-step sampling latency as unsolved.

Why not just prompt a large multimodal model centrally? Centralization defeats the privacy constraint for face and voice data, which cannot be anonymized without destroying the cues these tasks need. Beyond that, Friends-MMC reports 46–61% speaker identification for general multimodal models, and Chan et al. report near-random performance on distant links.

Decisive Checks

Three results would settle the proposal.

The first is the stratified gain curve over flat attention fusion on MELD weighted F1 and Friends-MMC response accuracy, as a function of speaker count and non-adjacent reply rate. A rising curve supports the claim. A flat curve refutes it.

The second is structure accuracy on the hand-annotated multimodal addressee set, under three conditions: federated with the visual keys, federated without them, and centralized. This tells us whether cross-modal cues improve structure estimation and how much federation costs.

The third is the structure-only versus share-everything comparison under modality-skewed clients. If sharing only the small structure head matches or beats full sharing on clients that lack video, then conversational geometry is the part worth federating. That would give later privacy-attack evaluations a concrete, low-content target.