Protecting What Only One Modality Knows: Neighborhood-Conditioned Independence for Deep Multimodal Graph Networks

When a graph neural network (a model that updates each node by repeatedly averaging information from its neighbors) is run on a graph whose nodes carry both text and images, two separate forces push the representation toward the same impoverished state. Stacking many averaging layers makes all nodes look alike, which is called over-smoothing. Separately, the usual fix for the modality gap, the tendency of text and image embeddings to sit in separate regions of a shared space, is to pull the two modalities together. Our hypothesis is that these two forces destroy the same information: the part of a node's image that its text cannot explain, and vice versa. Graph edges in these datasets usually connect semantically similar items, so the meaning shared across modalities tends to vary smoothly over the graph, while modality-private detail varies sharply from node to node. Averaging filters out sharp variation. If this is right, deep multimodal graph networks lose their private signal first. Alignment losses and residual shortcuts do not specifically protect it.

We propose to test this hypothesis directly and then build a targeted fix. The fix splits each node into a shared stream that is allowed to smooth and a per-modality private stream. An adversarial critic, trained to predict the private stream from the neighborhood's aggregated shared representation and from the other modality, forces the private stream to hold only information that neither the neighbors nor the other modality can supply. That information is the part of the node that is its own. The contribution has two halves. The first is a measurement: how the rank of shared and private components collapses with depth. The second is a mechanism whose independence target is defined relative to the graph neighborhood rather than between fixed feature partitions.

Core Claim

Over-smoothing on multimodal attributed graphs removes cross-modally unexplained variance before it removes shared variance. A regularizer that makes a private stream independent of neighborhood-smoothed shared features preserves task-relevant node identity at depth better than three alternatives: residual shortcuts, ego-anchored contrastive learning, and alignment by vector quantization. The claim is falsifiable at both stages. If the private residual is not higher-frequency on the graph than the shared component, or if its effective rank does not fall faster with depth, the premise fails. If a two-stream model without the independence penalty matches the full model, the adversarial ingredient is unnecessary.

Relation to Prior Work

Two papers motivate the premise. Don't Lose Yourself (RedNnD) reports that adding visual and textual features to recommendation graph networks makes over-smoothing worse. Its remedy is to contrast each node's initial embedding with its layer-averaged neighbors using InfoNCE. That keeps the node close to its starting point but never separates what is shared from what is private. Revisiting Multimodal Emotion Recognition in Conversation from the Perspective of Graph Spectrum (GS-MCC) argues that low-pass aggregation erases high-frequency inter-modal complementarity in dialogue graphs. It splits frequency bands with Fourier operators and admits that deeper networks still collapse. Neither paper measures which information is lost, and neither enforces independence.

Two papers constrain the design. Are we measuring oversmoothing in graph neural networks correctly? shows that Dirichlet energy misses the accuracy drop, while effective rank (the entropy of the normalized singular values) and numerical rank track accuracy with an average Pearson correlation of 0.91. That result dictates our diagnostic. The same paper warns that high rank does not imply usefulness, since noise has maximal rank. We therefore pair every rank measurement with a probe for task-relevant private content. Toward Effective Multimodal Graph Foundation Model (PLANET) aligns images to text anchors with vector quantization and gates modality interaction. It is the strongest multimodal-graph system in the file, but it neither monitors rank collapse with depth nor protects modality-private variance. Our worry is that anchoring images to text accelerates exactly the loss we hypothesize.

Two lines of work supply the method. RoPAD uses cross-prediction critics with gradient reversal to scrub nuisance factors. Its ablation on GCT1 shows the error rising from 4.2% to 28.7% without the critic. DICE-FER minimizes mutual information by discriminating joint pairs from shuffled pairs. We reuse the cross-prediction form of the critic.

The closest existing combination is Independence Promoted Graph Disentangled Networks (IPGDN). It routes neighbors into latent channels and penalizes dependence between those channels with HSIC, a kernel dependence measure, reaching 84.1% on Cora. It differs from our proposal in three ways. Its channels are unimodal. Its independence is between parallel channels of the same node rather than between a node's private content and its neighborhood. It never measures rank at depth.

Explore More, Learn Better applies a variational mutual-information upper bound across parallel paths in a multimodal language model. That shows mutual-information minimization helps multimodal embeddings, but the setting has no graph and no smoothing.

Research Program

The first stage is diagnostic and needs no new model. For each modality $m$ with node features $X_m$ from frozen CLIP-style encoders, we fit a ridge regression from the other modality. This splits $X_m$ into a cross-modally explained part $S_m$ and a residual $R_m$. We measure the graph frequency of each part with the normalized Rayleigh quotient $\mathrm{tr}(Y^\top L Y)/\mathrm{tr}(Y^\top Y)$, where $L$ is the normalized graph Laplacian and $Y$ is $S_m$ or $R_m$. We then pass each part through untrained and trained graph networks of depth 2 to 32 and track effective rank and numerical rank separately for each. In parallel, we track a probe: how well a linear classifier on the depth-$k$ representation recovers labels that are predictable from $R_m$ but not from $S_m$. The prediction is threefold: $R_m$ has a higher Rayleigh quotient, its rank falls faster, and its probe accuracy collapses earlier.

The second stage builds the model. Each node $v$ has a shared stream $h_v$, which undergoes ordinary message passing, and a private stream $p_v^m$ for each modality. Each private stream is computed from the node's own features only, and at each layer is passed through a high-pass update or no aggregation at all. The shared stream carries a light cross-modal alignment loss. The private stream is trained to reconstruct the residual $R_m$, which prevents it from collapsing to zero. A critic $q$ receives the mean of the neighbors' shared states and the other modality's features, and tries to predict $p_v^m$. The encoder is trained with gradient reversal to defeat it:

$$ \mathcal{L} = \mathcal{L}_{\text{task}} + \lambda_{\text{rec}} \lVert p_v^m - R_{m,v} \rVert^2 - \lambda_{\text{adv}} \min_q \mathbb{E}\,\big\lVert p_v^m - q\big(\bar h_{\mathcal{N}(v)}, x_v^{m'}\big) \big\rVert^2 . $$

Here $\bar h_{\mathcal{N}(v)}$ is the neighbor-averaged shared state, $x_v^{m'}$ is the other modality's input, and the $\lambda$ weights balance the terms. The task head reads the concatenation of $h_v$ and all $p_v^m$.

What is new is the conditioning set of the independence constraint. The constraint does not require two parallel channels to be independent. It requires that the node's private content cannot be recovered from its neighborhood or from its other modality. That is an operational definition of information that averaging cannot rebuild. To address training instability, we compare three critics of increasing strength: a linear critic, which amounts to a decorrelation penalty; a two-layer MLP; and an HSIC penalty in place of any critic. This lets the experiments decide whether nonlinearity is needed.

Baselines

The naive reference is a plain GCN on concatenated frozen text and image features at depths 2, 4, 8, 16 and 32. It fixes the collapse curve that every method should flatten.

The skip-connection baseline adds an initial-residual update, $H^{(l+1)} = (1-\alpha)\tilde A H^{(l)} + \alpha H^{(0)}$, in the style of GCNII and APPNP. Here $\tilde A$ is the normalized adjacency matrix, $H^{(l)}$ the layer-$l$ node representations, and $\alpha$ the residual weight. Because it re-injects the private signal at every layer, it is the most direct competitor to our private stream. The spectral baseline is the high-pass filterbank of Complete the Missing Half, which reports roughly 10% gains on Texas and Cornell over a plain GCN. The contrastive baseline is RedNnD's ego-anchor InfoNCE, carried over to node classification.

The strongest existing multimodal-graph method is PLANET. The file reports a relative 15.68% gain for it in few-shot node classification on MAGB and MM-GRAPH over UniGraph2, but gives no absolute accuracy. For absolute accuracy on those datasets, no reported result was found on Lacuna, so we will rerun PLANET with its released protocol. For recommendation, the strongest reported reference is RedNnD's relative gain of 10.78% in Recall@10 and 10.68% in NDCG@10 on Amazon Office. Again, no absolute Recall@10 was found on Lacuna. On IEMOCAP, GS-MCC reports +3.3% W-Acc and +3.2% W-F1 over AdaGIN.

The ablations isolate the key ingredient. The first is a two-stream model with $\lambda_{\text{adv}}=0$. The second replaces the neighborhood conditioning with a standard shared-private independence penalty, as in IPGDN, applied between channels of the same node. The third swaps the adversarial critic for HSIC.

All methods use the same frozen encoders, splits and depth grid. Each gets the same budget of 40 hyperparameter trials with the same early-stopping rule, and results are averaged over five seeds. We count a win only under one condition: at depth 8 or greater, the proposal must beat the best of the residual baseline and PLANET by more than two seed standard deviations. At depth 2 it must stay within noise of the best method.

Why not just…?

Why not just use initial-residual connections? This is the real threat. Re-injecting $H^{(0)}$ keeps private information available at every depth, and it may be enough. It does not stop the task head from depending on the shared component, which smooths away. It also does not stop the re-injected signal from being dominated by the part that neighbors already supply. Are we measuring oversmoothing in graph neural networks correctly? reports rank collapse tracking accuracy across architectural modifications, but that does not settle our case. The deciding test is the residual-private probe: if GCNII-style residuals keep that probe flat with depth and match our accuracy, we will report that the independence term is unnecessary.

Why not just contrast each node with its initial embedding, as RedNnD does? Contrastive learning raises agreement between views. It never removes shared content from a stream, so the ego anchor can be satisfied entirely through the shared component. The ReViSE results in Learning Robust Visual-Semantic Embeddings show pairwise matching leaves distributions misaligned. More directly relevant, the DICE-FER comparison reports contrastive baselines leaking confounders that adversarial minimization removes. Our experiment compares both methods on the residual probe: if RedNnD's gains come from shared content, its probe collapses while ours does not.

Why not just use HSIC or a linear decorrelation penalty instead of an adversary? These are more stable, and at mini-batch scale on MAGB and MM-GRAPH their quadratic cost is affordable. We therefore do not assume the adversary wins. Both are run as ablations under the same conditioning set. If the linear or HSIC variant matches the MLP critic, we adopt the simpler one, and the contribution becomes the neighborhood-conditioned target rather than the adversarial game.

Why not just use frozen CLIP features with a two-layer GNN? Shallow networks avoid collapse but cannot see far into the graph. PLANET and UniGraph2 both report gains from structure over static CLIP features. The depth-2 point of our naive baseline is this alternative. If no method beats it at depth 8 or more on any dataset, the case for going deep fails, and we would report that.

Proposed Experiments

Node classification uses MAGB (Movies, Toys, Grocery, Reddit-S) and MM-GRAPH (Amazon-Sports, Amazon-Cloth, Goodreads-NC), in both full-label and few-shot splits. Recommendation uses Amazon Baby, Sports, Clothing and Office with Recall@10 and NDCG@10. The dialogue graph setting uses IEMOCAP with W-Acc and W-F1. At every depth we record accuracy, effective rank and numerical rank for the shared and private components separately, the ridge-regression cross-modal prediction score as a measure of the modality gap, and the residual-private probe.

Three checks decide the project. The first is the stage-one diagnostic on plain GCNs: the residual must be higher-frequency than the shared component, and its effective rank and probe accuracy must fall earlier with depth. If this fails on MAGB, the project stops or is rescoped. The second is the ablation with $\lambda_{\text{adv}}=0$ against the full model at depths 8 to 32. It shows whether independence, rather than architecture alone, preserves the private probe and accuracy. The third tests whether closing the modality gap speeds up collapse. We sweep the shared-stream alignment weight and compare how quickly effective rank declines with and without the private stream. This answers the unresolved tension between alignment and diversification that neither the alignment literature nor the over-smoothing literature has tested.