Abstract
Masked graph autoencoder (MGAE) has emerged as a promising self-supervised graph pre-training (SGP) paradigm. However, existing methods typically perform masking and reconstruction in the raw data space, similar to techniques in computer vision (CV) and natural language processing (NLP). This approach often fails for graphs due to their non-Euclidean properties and unstable local connection structures, which introduce significant uncertainty in inferring masked data. To address this, we propose Robust mAsked gRaph autoEncoder (RARE). RARE improves the reliability of self-supervision by masking and reconstructing node samples in a high-order latent feature space. Our joint strategy—operating in both latent and raw data spaces—yields improved stability and performance across diverse downstream tasks.
Introduction: The Challenge of Graph Masking
Masked Autoencoders (MAEs) have revolutionized CV and NLP by hiding data segments (like image patches or words) and requiring the model to reconstruct them. While effective for Euclidean data, applying this directly to graphs is problematic. Unlike images, where spatial proximity provides strong context, or text, where grammar guides reconstruction, graphs are non-Euclidean. Local connections in social or biological networks are often "unstable"; masking a single node can create massive structural ambiguity.
As shown in the figure above, while image and text masking rely on predictable local structures, graph node/edge masking must contend with irregular connectivity. If a model only reconstructs low-level raw attributes from these unstable neighborhoods, it generates unreliable self-supervision signals, leading to inferior representations.
The RARE Framework
RARE introduces a Masked Latent Feature Completion (MLFC) scheme designed to capture high-order correlations that are difficult to observe in raw data. The framework operates through three primary stages:
- Dual Masking: The input graph is partitioned into visible and masked nodes.
- Latent Feature Prediction:
- An Online Graph Encoder processes visible nodes to generate representations.
- A Latent Feature Predictor estimates the latent representations of the masked nodes based on the visible ones.
- A Momentum Graph Encoder (a slowly moving average of the online encoder) processes the raw data of masked nodes to provide a stable target for the predictor.
- Joint Reconstruction: The model optimizes two losses: a Latent Matching Loss ($L_M$) to align predicted features with the momentum encoder's output, and a Raw Attribute Reconstruction Loss ($L_R$) to map latent features back to original node attributes.
Experimental Evaluation
We evaluated RARE across 17 datasets spanning node classification (e.g., Cora, Flickr, Yelp), graph classification (e.g., IMDB, PROTEINS), and image classification treated as graphs (e.g., MNIST).
Key Findings
- Superior Performance: RARE consistently outperformed state-of-the-art competitors like GraphMAE. On large-scale datasets like Flickr and Yelp, it improved accuracy by 7.7% to 14.8% over supervised baselines.
- Robustness to Noise: RARE demonstrates significant resilience to outliers. While competing models show sharp performance drops when 5% of node features are shuffled, RARE maintains high accuracy and training stability.
- Latent Stability: Theoretical and empirical analyses (via T-SNE) confirm that latent space matching acts as a powerful regularizer, helping the model differentiate between nodes that share common neighbors but belong to different categories.
Conclusion and Future Work
RARE advances self-supervised graph pre-training by moving beyond simple raw data reconstruction into high-order latent feature completion. This approach provides a more certain signal for learning in non-Euclidean spaces. Future research will explore extending RARE to capture global structural features beyond immediate local neighborhoods and applying the framework to data-incomplete scenarios where information is missing from the outset.
Summary updated