Many real graphs attach more than one kind of data to each node. A product in a co-purchase network has a title and a photo, and a Reddit post has a caption and an image. The usual recipe is to encode each modality with a pretrained model, glue the vectors together ("early fusion by concatenation"), and run a graph neural network on top. Benchmarks show that this recipe breaks in a specific way. Which modality matters changes from domain to domain: text dominates in e-commerce graphs and images dominate in Reddit graphs. When labels are scarce, the fused vectors can do worse than a single modality used alone. This proposal asks whether the graph itself can tell us, for each node and without any labels, which modality to trust.
The idea is simple. For each node, take the nodes it is strongly connected to through many-step random walks, and check, separately for text and for image features, how similar the node is to those structural neighbors. If a node's image looks like the images of its structural neighbors but its text does not, the image is the modality that agrees with the graph, and it should carry more weight in that node's fused representation. These per-node weights are computed before any classifier is trained. Gradient descent cannot push them toward whichever modality is easiest to fit, which is the mechanism that, by our reading, drives modality dominance and collapse in learned fusion.
Core Claim
Early fusion fails on multimodal graphs because its mixing weights are either fixed (concatenation treats every node the same) or learned from labels (gates that collapse onto the easiest modality and overfit when labels are few). The timing of fusion is not the root problem. Our claim is that weights derived from agreement between each modality and the graph's long-range structure fix both failures. They are label-free, so they survive few-shot training. They vary per node, so they can track heterogeneous modality quality inside one graph. They also make fusion non-additive, because each node's weight on text depends on its image features as well as its text features. The research question is whether this cheap, label-free signal matches or beats learned fusion when labels are scarce, and whether its per-node variation carries information that a single dataset-level modality weight does not.
Where This Sits
Three bodies of work feed this proposal. They play different roles.
The motivating evidence comes from multimodal fusion studies. Two systems working on Dravidian-language data found that concatenating modalities scored below the best single modality. On Tamil troll memes, the best concatenated model reached weighted F1 0.49 against 0.52 for text alone (CUET-NLP@DravidianLangTech-ACL2022). On Tamil hate speech, an audio-text system reached F1 0.25 against 0.35 for audio alone (DLRG@DravidianLangTech 2025). Multimodal Compact Bilinear Pooling explains part of the reason: concatenation followed by a linear layer is purely additive, so it cannot express one modality conditioning another. On graphs specifically, the MAGB benchmark (When Graph Meets Multimodal) and OpenMAG document domain-dependent dominance and few-shot vulnerability of fused features. These are exactly the two failures our weights target.
The constraining evidence is MuMo. In molecular property prediction it found that injecting structure late beat early fusion by 14–17%, because early fusion collapsed onto one noisy modality. That result seems to argue against any early fusion. We take it head on. MuMo's early-fusion variant learns its mixing by gradient. Our hypothesis is that collapse comes from that learning, not from fusing early as such. A post-propagation late-fusion baseline is therefore required, and if it matches our method, the hypothesis is wrong.
The methods we reuse come from topology-aware embedding. RWR-Reg shows that random-walk-with-restart profiles carry structural information beyond what local message passing captures. It also shows that keeping only the top-K entries of each profile preserves most of the gain, which makes our agreement score affordable. NRP shows that raw personalized PageRank is biased toward dense hub neighborhoods and corrects this by reweighting nodes by degree. We borrow that correction so that hubs do not dominate the agreement computation. Community-prior methods such as CNRL and Modularity-Aware GAEs argue that mesoscopic structure is a meaningful semantic context, and that is the premise behind treating structural neighbors as a reference for modality quality.
The closest competitors attack the same problem with learned machinery. LION propagates text and image features through a Clifford-algebra geometry and then fuses them with a learned energy-aware gate. It reports an average 5.84% node-classification margin over NTSFormer and UniGraph2 across nine text-image graph datasets. UniGraph2 routes modality features through learned mixture-of-experts layers and adds a shortest-path decoder that injects global structure. Graph4MM uses PageRank-diffused attention to carry multi-hop structure into a vision-language model. NTSFormer handles nodes that arrive with no edges or with missing modalities. In all four, the graph tells the model where to propagate information, while trained parameters decide how much each modality counts. In our proposal the graph also decides how much each modality counts, per node and before training, and we test whether this matters most in exactly the low-label regime where learned gates are weakest.
Research Program
The method has three steps. All modality encoders stay frozen; we use CLIP image and text features so that comparisons with MAGB and OpenMAG are on equal footing.
The first step computes a sparse structural profile for each node. We run random walk with restart with degree correction in the style of NRP, keep the top-K entries, and renormalize them into weights $\tilde r_{ij}$ over a structural neighbor set $S_i$. The second step scores how well each modality $m$ agrees with that profile:
$$a_i^m = \frac{1}{\sigma^m}\Big(\sum_{j \in S_i} \tilde r_{ij}\, \cos(x_i^m, x_j^m) - \mu^m\Big),$$
where $x_i^m$ is node $i$'s frozen feature in modality $m$. The quantities $\mu^m$ and $\sigma^m$ are the mean and standard deviation of cosine similarity between random node pairs in that modality. Subtracting them matters because pretrained embedding spaces differ in how similar unrelated items look on average, and without the correction the two modalities' scores would not be comparable. The cost is $O(NK)$ similarity evaluations per modality.
The third step builds the fused node input as
$$z_i = \big[\, w_i^{1}\, (\Pi X^{1})_i \,;\; w_i^{2}\, (\Pi X^{2})_i \,\big], \qquad w_i^m = \mathrm{softmax}_m\!\left(a_i^m/\tau\right),$$
where $\Pi$ is the same sparse diffusion operator applied to each modality's feature matrix $X^m$, so that features are smoothed over structural neighbors before fusion, and $\tau$ is a temperature. A two-layer classifier is then trained on $z$. The weights $w_i^m$ are never updated by the classification loss. Because each $w_i^m$ depends on both modalities through the softmax, the classifier's input already contains a node-conditional interaction that plain concatenation cannot represent.
Beyond accuracy, the program analyzes what the weights encode. We will measure how much $w_i$ varies within each dataset; it is per-node variation that justifies a per-node method. We will check whether nodes with high image weight are the ones where text is short or generic, as MAGB's explanation of Reddit graphs predicts. And we will check how the weights respond when the graph or one modality is deliberately damaged.
Baselines
The naive reference points are a two-layer classifier on the concatenated frozen CLIP features with no graph, and a graph convolutional network on the same concatenation. These are the standard early-fusion recipe that MAGB and OpenMAG show to be brittle. Alongside them we run the best single modality with the same diffusion operator, because a multimodal method that cannot beat it has failed by the standard set in the Dravidian studies. All baselines use MAGB's splits and frozen encoders, ten seeds, and the same hyperparameter search budget of 30 trials per method per dataset, on the same GPU budget.
The baseline that isolates our key ingredient is the identical pipeline with the weights replaced by a per-node gate. The gate is a small network over the propagated features of both modalities, trained end to end with the classification loss. It shares architecture, diffusion, and budget with our method; only the source of the weights differs. If topology-derived weights do not beat this learned gate in the few-shot settings, our central claim fails. A second ablation replaces per-node weights with a single dataset-level weight chosen by validation accuracy. This is the "just pick the better modality" oracle, and it tests whether per-node variation adds anything. A third ablation uses raw random-walk profiles without degree correction, to check whether hub bias matters as NRP suggests.
The strongest existing method is LION. The file reports its result only as a relative figure: an average 5.84% node-classification margin over NTSFormer and UniGraph2 across nine datasets, including Movies, Grocery, RedditS, and Goodreads. No absolute per-dataset accuracies for LION, Movies, Grocery, or RedditS were found on Lacuna. No public LION code is listed either, so we will reimplement it from the paper and first confirm that the reimplementation reproduces its reported margin over UniGraph2, for which code exists (UniGraph2 codebase). For cold-start nodes we include NTSFormer, which reports 61.58% accuracy on Goodreads-NC against 55.56% for an MLP and 54.01% for GLNN. We also include MAGB's strategy of replacing CLIP features with LLaMA-3.2-11B Vision node embeddings, which MAGB reports improves GNN accuracy by 2.95% on average over visual-only embeddings. Beating LION means two things. In 1-, 3-, and 5-shot settings, our method must exceed it on a majority of datasets with non-overlapping 95% confidence intervals across seeds. Under full supervision, it must come within one accuracy point of LION at a small fraction of its training cost. Matching LION only under full supervision would make this a cheaper method, not a better one, and we would report it that way.
Why not just…?
Why not just train a per-node gating network, as LION and UniGraph2 effectively do? With ample labels, a learned gate may well match or beat fixed topology weights, and we treat that as a real threat in the full-supervision regime. The case against it is in the low-label regime. MAGB and OpenMAG report that fused high-dimensional features hurt GNNs in few-shot settings. The meme and hate-speech results show learned fusion sliding toward one modality even when labels are moderately plentiful. The learned-gate ablation above settles the question directly. If the gate matches us at 1 to 5 shots, the proposal's main claim is refuted.
Why not just pick the better modality per dataset on a validation set? MAGB's finding of text dominance in e-commerce and image dominance in Reddit could be fully explained by a single number per dataset. If so, per-node weights are unnecessary decoration. The dataset-level oracle baseline tests this, and so does the within-dataset variance analysis. If the oracle ties our method and the weights barely vary across nodes, we will report that the useful signal is dataset-level and shrink the claim to "topology picks the right dataset-level weight without labels." That would still remove the need for a labeled validation set, which matters in few-shot settings.
Why not just fuse late, after separate propagation, as MuMo's result suggests? MuMo's 14–17% gain from late injection is the strongest evidence that fusion timing matters. Post-propagation concatenation is cheap and includes no gate at all. It is one of our baselines, and it also corresponds to our method with $w_i^m$ fixed at one half. If it matches us, then the weights add nothing and MuMo's interpretation is correct for graphs too. If our method wins, especially when one modality is corrupted, then the weighting is doing the work and MuMo's collapse is better explained by gradient-learned mixing.
Why not just use a joint vision-language encoder so that modalities are already aligned? MAGB's LLaMA-3.2-Vision embeddings help, but they cost a large model forward pass per node and still produce one fixed vector per node, with no way to signal that a particular node's image is unreliable. We include those embeddings as a baseline. We also compute agreement weights on top of them, treating the VLM embedding as a third modality. Since this method's cost does not grow exponentially with the number of modalities, as LION's does, adding a third is straightforward.
Proposed Experiments
The main evaluation uses MAGB's Movies, Grocery, Toys, RedditS, and Reddit-M, together with Goodreads and Ele-fashion from OpenMAG and LION's suite. Node classification is run at 1, 3, 5, and 20 shots per class and under full supervision, with accuracy and macro-F1 reported over ten seeds. A secondary run on OpenMAG's video-recommendation graphs, which carry video, audio, and text, tests whether the method extends past two modalities. No reported result was found on Lacuna for these graphs, so we report our own baselines there.
Three checks decide whether the mechanism is real. The first is controlled corruption. On a random 30% of nodes, we swap images with those of random other nodes. The image weights on exactly those nodes should fall, and accuracy should degrade much less than for concatenation or the learned gate. If the weights do not single out corrupted nodes, the agreement score is not measuring modality reliability. The second is edge shuffling. When edges are rewired at random, the agreement scores lose their meaning, the weights should flatten toward one half, and performance should fall to the level of post-propagation concatenation rather than below it. A drop below that level would mean the method is fragile on graphs whose structure is unrelated to the labels, and heterophilous graphs are the obvious risk. The third is the heterogeneity analysis. We will report how much the weights vary within each dataset and how the gap between our method and the dataset-level oracle grows as that variation grows. The per-node claim survives only if this relationship is positive.
Isolated cold-start nodes have no structural neighbors, so no agreement score can be computed for them. For those nodes we evaluate a fallback that borrows weights from feature-space nearest neighbors, compared against NTSFormer's reported 61.58% on Goodreads-NC. We present this as a stated limitation with a partial fix, not as a central contribution.
References
- CUET-NLP@DravidianLangTech-ACL2022: Investigating Deep Learning Techniques to Detect Multimodal Troll Memes
- DLRG@DravidianLangTech 2025: Multimodal Hate Speech Detection in Dravidian Languages
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- When Graph Meets Multimodal: Benchmarking and Meditating on Multimodal Attributed Graph Learning
- OpenMAG: A Comprehensive Benchmark for Multimodal-Attributed Graph
- Structure-Aware Fusion with Progressive Injection for Multimodal Molecular Representation Learning
- The Impact of Global Structural Information in Graph Neural Networks Applications
- Homogeneous Network Embedding for Massive Graphs via Reweighted Personalized PageRank
- A Unified Framework for Community Detection and Network Representation Learning
- New Frontiers in Graph Autoencoders: Joint Community Detection and Link Prediction
- UniGraph2: Learning a Unified Embedding Space to Bind Multimodal Graphs
- Graph4MM: Weaving Multimodal Learning with Structural Information
- NTSFormer: A Self-Teaching Graph Transformer for Multimodal Cold-Start Node Classification