Understanding and Tackling Over-Dilution in Graph Neural Networks

Junhyun Lee, Veronika Thost, Bumsoo Kim, Jaewoo Kang, Tengfei MaSubmission to ICLR 2024 · Rejected

Abstract

Message Passing Neural Networks (MPNNs) have become the predominant architecture for representation learning on graphs. While they hold promise, several inherent limitations have been identified, such as over-smoothing and over-squashing. Both theoretical frameworks and empirical investigations substantiate these limitations, facilitating advancements for informative representation. In this paper, we investigate the limitations of MPNNs from a novel perspective. We observe that even in a single layer, a node's own information can become considerably diluted, potentially leading to negative effects on performance. To delve into this phenomenon in-depth, we introduce the concept of *Over-dilution* and formulate it with two types of dilution factors: *intra-node dilution* and *inter-node dilution*. *Intra-node dilution* refers to the phenomenon where attributes lose their influence within each node, due to being combined with equal weight regardless of their practical importance. *Inter-node dilution* occurs when the node representations of neighbors are aggregated, leading to a diminished influence of the node itself on the final representation. We also introduce a transformer-based solution, which alleviates over-dilution by merging attribute representations based on attention scores between node-level and attribute-level representations. Our findings provide new insights and contribute to the development of informative representations.
BibTeX
@inproceedings{lee2025understanding,
  author = {Junhyun Lee and Veronika Thost and Bumsoo Kim and Jaewoo Kang and Tengfei Ma},
  title = {{Understanding and Tackling Over-Dilution in Graph Neural Networks}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2025},
}
Figure