Improving Node Representation by Boosting Target-Aware Contrastive Loss

Ying-Chun Lin, Jennifer NevillearXiv 2024 · DOI · arXiv

Introduction

Graph representation learning aims to map complex nodes into low-dimensional embeddings that preserve essential structural and attribute information. While supervised methods excel when labels are abundant, they often overfit on small datasets. Conversely, self-supervised learning (SSL) captures broad patterns but remains "task-blind," potentially emphasizing graph signals irrelevant to the final downstream objective. This paper introduces Task-Aware Contrastive Learning (XTCL), a framework designed to bridge this relevance gap by adaptively selecting positive pairs based on task-specific importance.

The Relevance Gap in Graph SSL

A fundamental challenge in graph machine learning is that different tasks rely on different graph signals. For instance, node classification in a citation network might depend heavily on local topology, whereas product recommendation might prioritize node attributes. Standard SSL treats all signals uniformly, leading to suboptimal representations that may focus on "noise" relative to the target task.

The model space conceptualization: XTCL aims to navigate the potential model space to find the "Optimal Model" by weighting signals like topology and attributes based on task relevance.

Methodology: Task-Aware Contrastive Loss (XTCL)

The core of the proposed framework is the XGBoost Sampler (XGSampler). Unlike traditional contrastive learning which uses fixed heuristics for positive sampling, XTCL learns to identify high-quality positive examples by evaluating various "semantic relations" such as Jaccard similarity, PageRank, and attribute distributions.

XGSampler and Boosting

The XGSampler utilizes a small amount of labeled data to train an ensemble of relation regression stumps. By maximizing the mutual information between the learned representations and the downstream task labels, the sampler assigns global importance weights to different graph signals. This adaptive sampling ensures that the contrastive loss pulls together nodes that are truly relevant to the task, even for unlabeled nodes in a semi-supervised setting.

Empirical Evaluation

The authors evaluated XTCL on five benchmark datasets, including citation networks (Cora, CiteSeer, PubMed) and product networks (Amazon Photo, Amazon Computers).

Performance and Data Efficiency

XTCL consistently outperformed state-of-the-art supervised and self-supervised models. Notably, the method demonstrates superior data efficiency, maintaining a significant performance lead even when training labels are scarce. On the Cora dataset, XTCL achieved a top-tier accuracy of 85.19%, outperforming popular SSL methods like DGI and SUGRL. In link prediction tasks, XTCL reached an AUC of 97.13% on CiteSeer, a marked improvement over the baseline GCN.

Learning curves on CiteSeer show that XTCL (orange) consistently achieves higher accuracy than standard Supervised Contrastive Loss (SCL) and other baselines across varying training sizes.

Robustness and Interpretability

A key advantage of XTCL is its interpretability. By examining the weights assigned by XGSampler, researchers can identify which graph signals drive performance for a specific task. Furthermore, semi-synthetic experiments demonstrated that XTCL is more robust to noise than traditional GCNs, maintaining stable performance even when label-signal correlations are perturbed.

Conclusion and Limitations

XTCL effectively addresses the task-blindness of traditional graph SSL. While the framework introduces some computational overhead—with a worst-case complexity of $O(|V|^2)$ for dense relations—the authors provide GPU-accelerated implementations to mitigate this. Future work includes extending the framework to heterogeneous graphs and developing purely unsupervised task-awareness mechanisms.

Summary updated

Abstract

Graphs model complex relationships between entities, with nodes and edges capturing intricate connections. Node representation learning involves transforming nodes into low-dimensional embeddings. These embeddings are typically used as features for downstream tasks. Therefore, their quality has a significant impact on task performance. Existing approaches for node representation learning span (semi-)supervised, unsupervised, and self-supervised paradigms. In graph domains, (semi-)supervised learning often only optimizes models based on class labels, neglecting other abundant graph signals, which limits generalization. While self-supervised or unsupervised learning produces representations that better capture underlying graph signals, the usefulness of these captured signals for downstream target tasks can vary. To bridge this gap, we introduce Target-Aware Contrastive Learning (Target-aware CL) which aims to enhance target task performance by maximizing the mutual information between the target task and node representations with a self-supervised learning process. This is achieved through a sampling function, XGBoost Sampler (XGSampler), to sample proper positive examples for the proposed Target-Aware Contrastive Loss (XTCL). By minimizing XTCL, Target-aware CL increases the mutual information between the target task and node representations, such that model generalization is improved. Additionally, XGSampler enhances the interpretability of each signal by showing the weights for sampling the proper positive examples. We show experimentally that XTCL significantly improves the performance on two target tasks: node classification and link prediction tasks, compared to state-of-the-art models.
BibTeX
@misc{lin2024improving,
  author = {Ying-Chun Lin and Jennifer Neville},
  title = {{Improving Node Representation by Boosting Target-Aware Contrastive Loss}},
  year = {2024},
  eprint = {2410.03901},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.03901},
}
Figure