Generalised f-Mean Aggregation for Graph Neural Networks

Ryan Kortvelesy, Steven D. Morad, Amanda Prorok

Introduction

In Graph Neural Networks (GNNs), the aggregation step—how a node summarizes information from its neighbors—is a critical but often overlooked component. While standard operators like mean, sum, or max are common, they are inherently lossy and may not be optimal for all tasks. Researchers from the University of Cambridge introduce GenAgg, a learnable aggregation operator based on the generalized $f$-mean. GenAgg can adaptively represent a wide range of functions, minimizing information loss and improving GNN performance.

The Aggregation Bottleneck

Aggregation must be permutation-invariant and handle varying neighborhood sizes. Standard choices are simple but restrictive. Learnable alternatives like Deep Sets or SoftmaxAgg often struggle with sample efficiency or the inability to represent basic functions like standard deviation accurately. GenAgg addresses these issues by providing a flexible, numerically stable framework.

GenAgg: Learnable f-Mean Aggregation

GenAgg is built upon the generalized $f$-mean: $$f^{-1}\left(\frac{1}{n} \sum f(x_i)\right)$$

To enhance its capabilities, the authors propose the Augmented $f$-Mean (AFM), introducing two learnable parameters:

  1. Cardinality Dependence ($\alpha$): Allows the aggregator to transition between "mean-like" ($\alpha=0$) and "sum-like" ($\alpha=1$) behavior.
  2. Centralized Moments ($\beta$): Enables the aggregator to compute "centralized" features like variance or standard deviation by subtracting a portion of the mean before applying $f$.

Instead of a fixed mathematical form, $f$ and $f^{-1}$ are parameterized by small MLPs. An inverse loss term ensures they maintain their relationship, allowing the model to discover the optimal transformation for the data.

Experimental Setup

The effectiveness of GenAgg was evaluated through:

  • Aggregator Regression: Mimicking 13 standard aggregators (e.g., geometric mean, $L_2$ norm).
  • GNN Benchmarks: Node classification (CLUSTER, PATTERN) and image-to-graph classification (MNIST, CIFAR10) from the Benchmarking GNNs suite.
  • Small-scale studies: Evaluations on CORA, PubMed, and CiteSeer to assess overfitting.

Key Results

GenAgg demonstrates superior performance and flexibility across diverse tasks:

  • Universal Mimicry: It achieved a correlation of >0.96 with all 13 target aggregators in regression tasks, far outperforming other learnable methods.
  • State-of-the-Art Accuracy: GenAgg-based models consistently outperformed baselines like PNA, mean, sum, and max across all major benchmarks. For example, on CLUSTER, GenAgg achieved 0.627 accuracy compared to 0.602 for the standard mean.
  • Efficient Convergence: Training plots indicate that GenAgg often reaches higher accuracy levels much faster than fixed aggregators.
Performance comparison of GenAgg against standard and learnable aggregators across four benchmark datasets.

Discussion and Future Work

GenAgg offers explainability through its $\alpha$ and $\beta$ parameters, revealing whether a model prioritizes neighborhood size or feature variance. While it shows significant gains, its MLP-based implementation introduces some computational overhead compared to simple sum/mean operations. Future work may involve symbolic regression to derive faster, closed-form expressions post-training. Additionally, while highly effective on large benchmarks, care must be taken to avoid overfitting on very small datasets like CORA.

Summary updated

BibTeX
@inproceedings{kortvelesy2023generalised,
  author = {Ryan Kortvelesy and Steven D. Morad and Amanda Prorok},
  title = {{Generalised f-Mean Aggregation for Graph Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=JMrIeKjTAe},
}
Figure