Abstract
In the field of Graph Machine Learning, the "pre-train, fine-tune" paradigm often suffers from a "task gap" where pre-training objectives do not align with downstream requirements. This discrepancy is particularly pronounced in Heterogeneous Graphs (HGs), which involve complex node types and high-order relationships. This paper introduces CLEAR (Cluster-based prompt LEARning), a framework that bridges this gap by introducing virtual "cluster prompts." By reformulating classification tasks into graph reconstruction problems and utilizing structure-aware meta-path templates, CLEAR achieves significant performance improvements, particularly in few-shot scenarios.
Introduction
The standard approach for graph-based tasks involves training a model on massive unlabeled datasets to learn general structural patterns before fine-tuning on specific downstream goals. However, traditional pre-training tasks, such as predicting missing edges, often fail to capture the semantic nuances required for node classification in Heterogeneous Graphs.
Existing prompting methods typically attempt to align these tasks by modifying node features with learned vectors. However, in HGs—which contain diverse node types (e.g., authors, papers, venues) and complex meta-paths—feature-based prompting is insufficient. CLEAR addresses this by making prompting "structure-aware," treating prompts as physical components of the graph topology rather than mere feature modifications.
The CLEAR Framework
CLEAR employs a three-stage methodology to ensure that knowledge gained during pre-training transfers seamlessly to downstream applications.
1. Dual-View Pre-training
The process begins with a contrastive learning framework. CLEAR utilizes two distinct Graph Neural Networks (GNNs): a Structural GNN to capture local connectivity and a Semantic GNN to capture high-order relationships derived from meta-paths. The model is trained to align these two views, ensuring a robust representation of both the graph's layout and its underlying semantic logic.
2. Cluster Prompts as Virtual Nodes
Unlike traditional methods, CLEAR introduces virtual nodes called cluster prompts. Each virtual node represents a latent category or data cluster. This allows the model to reformulate node classification as a link prediction problem: determining the likelihood of a connection between a target node and a specific cluster prompt.
3. Meta-path Templates
To ensure prompts understand high-order semantics, CLEAR "injects" prompt tokens into existing meta-paths. For instance, a standard "Author-Paper-Author" path is extended into an "Author-Paper-Prompt-Paper-Author" template. A specialized discriminator, powered by a 1D-CNN, evaluates these generated paths to ensure they remain semantically valid within the context of the graph.
Experimental Evaluation
The authors evaluated CLEAR against state-of-the-art models using three benchmark datasets: ACM (citation network), DBLP (computer science bibliography), and IMDB (movie database).
Key Results
- Node Clustering: CLEAR demonstrated a 5% improvement in clustering metrics (NMI and ARI) on the ACM and DBLP datasets compared to previous pre-training models.
- Few-Shot Classification: In one-shot scenarios, CLEAR outperformed the next best prompting model (HetGPT) by 5.6% in Macro-F1 on the ACM dataset.
- Robustness: The framework proved exceptionally effective in low-data regimes, where the gap between CLEAR and traditional supervised models was most significant.
Conclusion
CLEAR represents a shift from feature-centric to structure-centric prompting in heterogeneous graph learning. By treating prompts as virtual nodes and integrating them into the graph's semantic meta-paths, the framework effectively bridges the gap between pre-training and downstream tasks. While future work may address scalability for industrial-sized graphs and automated meta-path discovery, CLEAR currently sets a new benchmark for performance and efficiency in few-shot graph learning.
Summary updated