Graph Representation Learning for Multi-Task Settings: a Meta-Learning Approach

Davide Buffelli, Fabio Vandin

TL;DR

This paper introduces a novel meta-learning approach for training Graph Neural Networks (GNNs) to produce general-purpose node embeddings that can be effectively used across multiple downstream tasks. Their method, SAME, achieves comparable or even superior performance to specialized single-task models, demonstrating the power of meta-learning for robust graph representation learning.

What Problem Does This Paper Address?

Graph Neural Networks (GNNs) have become a powerful tool for learning representations from graph-structured data. Typically, GNNs are trained in an end-to-end fashion for a specific task, resulting in highly specialized node embeddings. While this yields excellent performance on the target task, these embeddings often perform poorly when transferred to different tasks. This lack of transferability necessitates training separate models for each task, leading to significant computational overhead and redundancy, especially in scenarios where multiple tasks need to be performed on the same graph(s) (e.g., in social networks for user classification and link prediction). The paper addresses this challenge by aiming to learn a single GNN encoder that produces node embeddings generalizable across multiple tasks, thus avoiding the need for task-specific encoders.

Methods

The core of the proposed method is a meta-learning strategy called SAME (Single-Task Adaptation for Multi-Task Embeddings). This approach is inspired by optimization-based meta-learning techniques like MAML and ANIL. The key idea is to train a GNN encoder-decoder model using a "multi-task episode" design.

  • Meta-Learning Episodes: Each episode consists of multiple tasks (e.g., graph classification, node classification, link prediction). For each task within an episode, separate support (training) and target (validation) sets are created. This design mimics the classical training/validation split for single-task learning, encouraging generalization.
  • Inner Loop Adaptation: The GNN model is adapted to each task individually within the episode using a few gradient descent steps on the task's support set. This "single-task adaptation" in the inner loop is crucial, as it allows the model to learn to quickly adapt to individual tasks without the complexity of learning all tasks concurrently.
  • Outer Loop Update: The outer loop then updates the initial model parameters based on the performance of these individual adaptations on the respective target sets. The meta-objective aims to find parameters that allow for fast adaptation to multiple tasks.
  • Variants (iSAME and eSAME):
    • iSAME (Implicit SAME): All parameters of the model (backbone and task-specific heads) are adapted in the inner loop. This leverages MAML's implicit feature reuse.
  • eSAME (Explicit SAME): Only the task-specific decoder heads are adapted in the inner loop, while the backbone (encoder) parameters are only updated in the outer loop. This is inspired by ANIL and explicitly aims to learn general backbone features.
  • Final Encoder: After meta-training, the decoder heads are discarded, and the trained encoder GNN is used to generate the general-purpose node embeddings.

Datasets

The authors used datasets from the TUDataset library for training and evaluation:

  • ENZYMES: Protein structures belonging to six classes.
  • PROTEINS: Chemical compounds classified as enzyme or non-enzyme.
  • DHFR and COX2: Datasets of chemical inhibitors (active/inactive).

These datasets were chosen because they provide graph labels, node attributes, and node labels, making them suitable for the three main tasks considered: graph classification (GC), node classification (NC), and link prediction (LP).

Key Insights and Ideas

  • Meta-Learning for Generalization: The core insight is that by structuring meta-learning episodes to mimic single-task training/validation splits, the meta-learning procedure effectively optimizes for generalization across tasks.
  • Single-Task Adaptation is Key: The paper highlights that adapting to tasks individually in the inner loop, rather than trying to solve all tasks concurrently, is crucial for learning robust, generalizable representations. This bypasses the difficulty of concurrent multi-task learning.
  • Unexpected Benefit in Single-Task Settings: The meta-learning procedure, even when applied to a single task (ablated version), leads to embeddings that perform comparably or better than classical end-to-end trained models. This suggests that the episodic training strategy itself is a powerful way to learn better representations.
  • More Informative Embeddings: Embeddings learned via SAME lead to significantly higher performance on unseen tasks compared to embeddings from classically trained multi-task models, indicating they capture richer and more transferable information.

Key Results

  • Comparable or Better than Single-Task Models: Embeddings from SAME achieve performance comparable to, and often better than, specialized end-to-end trained single-task models across GC, NC, and LP tasks. In 50% of cases, SAME embeddings outperformed single-task models.
  • Superior to Classical Multi-Task Models: SAME-trained embeddings significantly outperform classically trained multi-task models, with a worst-case average drop in performance of less than 3% compared to 29% for classical methods. This demonstrates SAME's robustness to task interference.
  • Improved Performance on Unseen Tasks: When tested on tasks not seen during multi-task training, SAME embeddings led to at least 10% higher performance compared to classical multi-task models.
  • Ablation Study: The generalization-encouraging design of support/target sets was found to be the primary driver of SAME's success. The separate single-task adaptations in the inner loop improve learning efficiency by reducing the need for labels across all tasks simultaneously.

Limitations and Future Work

  • Computational Cost: While SAME aims to improve efficiency by producing general embeddings, the meta-learning training procedure itself can be computationally intensive.
  • Adaptation at Test Time: The current framework does not involve adaptation at test time, which is a common feature of meta-learning. The authors suggest exploring more advanced meta-learning strategies that might incorporate this.
  • Broader Applicability: The authors suggest that the model-agnostic nature of optimization-based meta-learning makes this approach adaptable to other domains beyond graph representation learning.
  • More Advanced Meta-Learning: Incorporating more advanced meta-learning strategies (e.g., those with implicit gradients) is mentioned as a future direction.

Summary updated

Abstract

Graph Neural Networks (GNNs) have become the state-of-the-art method for many applications on graph structured data. GNNs are a framework for graph representation learning, where a model learns to generate low dimensional node embeddings that encapsulate structural and feature-related information. GNNs are usually trained in an end-to-end fashion, leading to highly specialized node embeddings. While this approach achieves great results in the single-task setting, generating node embeddings that can be used to perform multiple tasks (with performance comparable to single-task models) is an open problem. We propose a novel representation learning strategy, based on meta-learning, capable of producing multi-task node embeddings. Our method avoids the difficulties arising when learning to perform multiple tasks concurrently by, instead, learning to quickly (i.e. with a few steps of gradient descent) adapt to multiple tasks singularly. We show that the embeddings produced by our method can be used to perform multiple tasks with comparable or higher performance than both single-task and multi-task end-to-end models. Our method is model-agnostic and task-agnostic and can hence be applied to a wide variety of multi-task domains.
BibTeX
@inproceedings{buffelli2022graph,
  author = {Davide Buffelli and Fabio Vandin},
  title = {{Graph Representation Learning for Multi-Task Settings: a Meta-Learning Approach}},
  booktitle = {IJCNN},
  year = {2022},
  url = {https://openreview.net/forum?id=ttgA2F23hZ},
}
Figure