When a model fuses text, audio, and video, one stream is often damaged: the microphone clips, the face turns away, or the speech recognizer garbles a word. Attention-based fusion does not handle this well. It decides how much each input counts by comparing inputs two at a time (a query against a key), and a corrupted token can still look similar enough to a query to receive weight. We propose a different rule for moments that are grouped together. Treat everything that happens in the same time span as one group, called a hyperedge (an edge that can join any number of items, not just two). Give each item in the group a weight based on how well the other modalities in that group predict it. Then send the group's consensus back into each item, sending more back to items that disagreed. Placed in the first layers of the encoders, this lets a clean modality repair a corrupted one before the damage spreads through the network.
The idea combines four lines of work, each with a specific job. Two-stage hypergraph message passing, where items first send information up to their groups and groups then send it back down, supplies the computation. Dynamic neighbor weighting supplies the principle that weights should depend on content rather than on graph degree. We replace its pairwise score with a score that only exists at the level of a group. Cross-modal attention appears as a special case: the consensus a node is scored against is built only from other modalities. Early fusion is where the module is placed, because On the Benefits of Early Fusion in Multimodal Representation Learning showed that fusing at the first layer lets a clean audio stream resolve badly corrupted digit images, while late fusion cannot. That result came from a recurrent convolutional model on toy data with no relational structure. We test whether it survives in a relational, attention-based fusion model on real conversational data.
Core Claim
The claim is narrow and falsifiable. On multimodal data where modalities are naturally grouped by co-occurrence, we compare two ways of weighting the members of a hyperedge. The first scores each member by leave-its-modality-out agreement. The second is the usual dot-product attention used by AllSetTransformer and hypergraph attention. We claim the first degrades more gracefully when one modality is corrupted at test time, and costs nothing on clean data. We also claim the benefit comes from placing the module early rather than late.
The weighting rule is as follows. For a hyperedge $e$ and a member node $v$ from modality $m(v)$, first compute a summary built only from members of other modalities:
$$s_{e,\neg m(v)} = \frac{1}{|M_e|-1}\sum_{m' \in M_e,\, m'\neq m(v)} \frac{1}{|e_{m'}|}\sum_{u \in e_{m'}} P_{m'} h_u .$$
Here $h_u$ is the feature of node $u$, $P_{m'}$ is a learned projection into a shared space for modality $m'$, $e_{m'}$ is the set of members of $e$ from modality $m'$, and $M_e$ is the set of modalities present in $e$. The summary averages within each modality first, so a modality with many tokens (video frames) cannot outvote one with few (an utterance).
The Stage 1 weight of $v$ in $e$ is
$$\alpha_{ve} = \operatorname{softmax}_{v\in e}\!\left(-\tfrac{1}{\tau}\,\lVert P_{m(v)}h_v - s_{e,\neg m(v)}\rVert^2\right),$$
where $\tau$ is a learned temperature. The hyperedge state is $h_e=\sum_{v\in e}\alpha_{ve}W_1h_v$, with $W_1$ a learned matrix.
Stage 2 writes the consensus back into each node through a residual gate. The gate $g_v$ is a small learned function of $v$'s own disagreement score, so nodes that disagree with their group receive more of the group's state. This gate is borrowed from the gated cross-attention of An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention, which mixes cross-modal context and original features with a learned gate to stop one modality from dominating. We condition that gate on group-level agreement rather than learning it as a free parameter.
The difference from dot-product attention matters. A corrupted token earns weight under dot-product attention whenever it happens to align with the query direction. Under consensus weighting it earns weight only if the other modalities actually agree with it. Pairwise attention cannot express this, because the score depends on the whole group minus the node's own modality.
Relation to Prior Work
Several papers already combine hypergraphs with multimodal attention. None of them studies how the member-weighting rule behaves under corruption.
Three papers motivate the gap. Variational Multi-Modal Hypergraph Attention Network for Multi-Modal Relation Extraction builds global, within-modality, and cross-modality hyperedges over text entities and detected objects. It runs multi-head node-hyperedge attention with Gaussian node features and reaches 85.22% F1 on MNRE. Its incidence structure is fixed and its attention is dot-product. HyperClaim learns continuous ("soft") incidence weights and calibrates text against video in both directions, reaching 83.7% on FakeSV. It is the closest work on dynamic weighting inside a hypergraph. Its weights are learned connection scores computed on top of a frozen vision-language model, not a cross-modal agreement test, and its fusion happens late. STMI forms hyperedges from cosine similarity between RGB, near-infrared, and thermal tokens, then applies cross-attention as a separate later block. We instead put the cross-modal comparison inside Stage 1.
Two papers supply the method. You are AllSet shows that two-stage hypergraph message passing is a composition of two learnable set functions and that earlier hypergraph networks are special cases. Our module is one such set function with a particular Stage 1 aggregator. AllSetTransformer, which aggregates with dot-product set attention, is therefore the natural ablation. HNHN supports keeping explicit hyperedge states, which we do.
One paper constrains the design. MATCH builds cross-modal hyperedges over conversational windows for emotion recognition. It reports collapse beyond four hypergraph layers from over-smoothing, where repeated averaging makes all node features alike. We therefore use one or two rounds with residual gates, not a deep stack.
Finally, Hypergraph Attention Networks for Multimodal Learning and SCAN both depend on external parsers (scene graphs or dependency trees) to define hyperedges. We use only timestamps, which the data already provides. We avoid learned hyperedge discovery on purpose. With learned queries forming the groups, the module becomes nearly indistinguishable from a latent-bottleneck cross-attention model, and the claim would stop being about the weighting rule.
Research Program
We will build a fusion module that sits after the first one or two layers of each unimodal encoder. In emotion recognition in conversation, the nodes are text sub-word tokens, audio frames, and video frames. Each utterance's time span defines one cross-modal hyperedge, and a sliding window of neighboring utterances defines a second, coarser hyperedge type. The rest of each encoder continues on the updated node features, and a standard classifier head predicts emotion. Aside from the Stage 1 aggregator and the gate, everything is shared with the ablations, so comparisons isolate the weighting rule.
There is an honest risk. Consensus weighting could suppress disagreement that is real signal, for example a sarcastic smile over negative words. To keep this information available to the classifier, we pass the per-node disagreement score forward as a feature and keep the original features in the residual path. We will also report results on the subset of utterances where annotators' emotion labels conflict with the text-only prediction. If our method loses there, the loss is a real cost and will be reported as such.
The analysis has three parts. First, we inject known corruption and measure how much Stage 1 weight lands on the corrupted tokens, compared with dot-product aggregation. Second, we trace how gate values distribute over corrupted and clean nodes. Third, we test placement by moving the same module to the last encoder layer, to check whether the early-fusion benefit reported in the C-LSTM study carries over.
Baselines
The naive reference point is late fusion. Each modality has its own encoder with pooled outputs, the pooled vectors are concatenated, and a classifier sits on top. It uses the same encoders, training data, and number of tuning trials as every other method. Any method that does not beat it under corruption tells us nothing.
The strongest published system on the chosen benchmarks is MATCH, which reports 72.55% weighted-F1 (72.64% accuracy) on IEMOCAP and 67.04% weighted-F1 (68.16% accuracy) on MELD (MATCH). We will cite those clean-data numbers directly. We will also rerun MATCH on our feature extractors and splits, with the same tuning budget, so it can face the corruption protocol. No reported result was found on Lacuna for MATCH, or any hypergraph fusion model, under test-time modality corruption. On clean data, beating MATCH means matching it within one standard deviation over five seeds; we do not claim a clean-data improvement. Under corruption, beating it means a higher area under the weighted-F1-versus-corruption-level curve, with non-overlapping seed intervals.
The strongest simple competitor is a joint-token early-fusion transformer. All text, audio, and video tokens are concatenated into one sequence at the same layer where our module sits, followed by ordinary self-attention. It is matched in parameters and floating-point operations and trained with the same corruption augmentation. This is the baseline most likely to erase our gain.
We also run bidirectional gated cross-attention in the style of the drug-target interaction model, at the same placement.
The key ablation is AllSetTransformer aggregation on exactly our hyperedges and placement (You are AllSet). It replaces only the consensus score with dot-product set attention. Further ablations switch off the disagreement-conditioned gate, replace timestamp hyperedges with random groups of equal size, and move the module to the final layer. All tuning uses the same number of trials per method on the validation split only.
Why not just…?
Why not just concatenate all tokens early and run self-attention? This is the real threat. A joint transformer can in principle learn to compare each token against other modalities. What is unclear is whether it does so under corruption it never saw, since softmax attention scores each key against a query one pair at a time. Our experiment trains both with identical noise augmentation, then tests on corruption types held out from training: frame dropping when trained with additive noise, and speech-recognition word swaps. If the joint transformer matches our degradation curves on the held-out corruptions, the consensus rule adds nothing, and we will report that.
Why not just use gated cross-attention? The gate in An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention is learned per feature, not per input. It can learn that a modality is usually weaker but cannot detect that this particular clip is corrupted. The per-sample weight-on-corrupted-tokens analysis tests this difference directly.
Why not just train with modality dropout or noise augmentation on a standard model? Augmentation is applied to every baseline, so any gain we report is on top of it. If augmentation alone closes the gap on held-out corruption types, the proposal fails.
Why not just use AllSetTransformer or hypergraph attention as is? They are the same computation with a pairwise score. The ablation isolates exactly that choice, so this alternative is part of the core experiment, not an argument around it.
Proposed Experiments
The first testbed is controlled. We use paired MNIST digits and Free Spoken Digit Dataset recordings, with white noise down to SNR 0.0625, following On the Benefits of Early Fusion in Multimodal Representation Learning. That paper reports only that early fusion beats late fusion at nearly all noise levels; no accuracy figures were found on Lacuna. Here, hyperedges pair each digit image with its utterance and its augmented views. This setting checks whether consensus weighting recovers the early-fusion advantage when the fusion model is attention-based rather than convolutional-recurrent.
The main testbed is IEMOCAP and MELD. At test time we corrupt one modality per utterance, chosen at random. The corruptions are Gaussian feature noise at five levels, video-frame dropout, audio clipping, and text word substitution at rates matched to speech-recognition error. We report weighted-F1 on clean data, the area under the degradation curve, and the worst-modality degradation. We also report the fraction of Stage 1 weight assigned to corrupted tokens, which is only measurable because we know where the corruption was injected.
The results are decisive in either direction. The idea holds if three things happen: consensus aggregation beats the AllSetTransformer ablation and the joint transformer on held-out corruption types, the gain vanishes when the module is placed late, and the gain vanishes when hyperedges are randomized. The idea is wrong if the joint transformer or augmentation alone matches those curves, or if clean-data weighted-F1 drops by more than one point relative to MATCH.
References
- On the Benefits of Early Fusion in Multimodal Representation Learning
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- Variational Multi-Modal Hypergraph Attention Network for Multi-Modal Relation Extraction
- HyperClaim: Fine-Grained Cross-Modal Hypergraph Reasoning for Video Misinformation Detection
- STMI: Segmentation-Guided Token Modulation with Cross-Modal Hypergraph Interaction for Multi-Modal Object Re-Identification
- You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks
- HNHN: Hypergraph Networks With Hyperedge Neurons
- MATCH: Modality-Calibrated Hypergraph Fusion Network for Conversational Emotion Recognition
- Hypergraph Attention Networks for Multimodal Learning
- Cross-Modal-Aware Representation Learning with Syntactic Hypergraph Convolutional Network for VideoQA