LATEX-GCL: Large Language Models (LLMs)-Based Data Augmentation for Text-Attributed Graph Contrastive Learning

Haoran Yang, Xiangyu Zhao, Sirui Huang, Qing Li, Guandong XuarXiv 2024 · OpenReview

Abstract

Graph Contrastive Learning (GCL) is a potent paradigm for self-supervised graph learning that has attracted attention across various application scenarios. However, GCL for learning on Text-Attributed Graphs (TAGs) has yet to be explored. Because conventional augmentation techniques like feature embedding masking cannot directly process textual attributes on TAGs. A naive strategy for applying GCL to TAGs is to encode the textual attributes into feature embeddings via a language model and then feed the embeddings into the following GCL module for processing. Such a strategy faces three key challenges: I) failure to avoid information loss, II) semantic loss during the text encoding phase, and III) implicit augmentation constraints that lead to uncontrollable and incomprehensible results. In this paper, we propose a novel GCL framework named LATEX-GCL to utilize Large Language Models (LLMs) to produce textual augmentations and LLMs' powerful natural language processing (NLP) abilities to address the three limitations aforementioned to pave the way for applying GCL to TAG tasks. Extensive experiments on four high-quality TAG datasets illustrate the superiority of the proposed LATEX-GCL method.

Introduction

Graph Contrastive Learning (GCL) has emerged as a powerful self-supervised technique, learning representations by comparing different "views" of the same data. However, its application to Text-Attributed Graphs (TAGs)—where nodes contain rich textual information—remains challenging. Traditional GCL methods typically rely on "feature masking," which randomly hides parts of a data vector. When applied to text embeddings, this approach often leads to significant information loss, semantic degradation, and uncontrollable results because the perturbations occur in an abstract mathematical space rather than the semantic space of natural language.

To address these issues, we introduce LATEX-GCL, a framework that leverages the semantic reasoning capabilities of Large Language Models (LLMs) to perform data augmentation directly on raw text. By rewriting, shortening, or expanding node text before it is processed by a graph neural network, LATEX-GCL preserves the underlying logic and meaning of the data, overcoming the limitations of traditional vector-based augmentation.

Methodology

The LATEX-GCL pipeline moves the augmentation step from the abstract embedding space back into the realm of natural language. The process consists of four primary stages:

  1. LLM-Based Augmentation: Instead of masking vectors, an LLM (e.g., GPT-3.5) modifies the raw text of each node using three strategies: Shorten (summarizing to filter noise), Rewriting (improving clarity to identify invariant semantics), and Expansion (adding context from the LLM’s internal knowledge base).
  2. Text Encoding: A smaller, efficient model (e.g., BERT or DeBERTa) converts both original and augmented text into feature embeddings.
  3. Graph Encoding: A Graph Convolutional Network (GCN) processes these embeddings alongside the graph's topological structure.
  4. Contrastive Learning: The model employs the InfoNCE loss function to ensure that the original node and its LLM-augmented version are mapped close together in the latent space, while other nodes are pushed away.
The LATEX-GCL framework architecture, illustrating the LLM-based augmentation and contrastive learning process.

Experiments and Results

The framework was evaluated on four large-scale TAG datasets derived from Amazon product categories: Books-Children, Books-History, Ele-Computers, and Ele-Photo. These datasets are significantly more text-rich than standard benchmarks used in prior graph research.

As shown in the results table below, LATEX-GCL variants consistently outperformed state-of-the-art baselines such as GraphCL, GCA, and GRACE. For example, on the Books-Children dataset, LATEX-GCL achieved an accuracy of 41.72%, a notable improvement over the best baseline (BGRL) at 37.99%. Furthermore, LATEX-GCL demonstrated superior scalability; while the popular GRACE method suffered from Out-of-Memory (OOM) errors on larger datasets, LATEX-GCL remained efficient.

Main experimental results comparing LATEX-GCL variants with baseline methods across four datasets.

Key Insights

  • Semantic Augmentation: Using LLMs to "rewrite" or "expand" text provides a more meaningful contrastive view than random feature dropping, as it preserves the underlying logic of the node.
  • Prompt Sensitivity: The "Expansion" strategy (E) excelled on book datasets where context is vital, while "Rewriting" (R) was highly effective for electronics datasets, likely due to the need to clean up noisy consumer reviews.
  • Encoder Choice: Bidirectional models like BERT proved to be superior text encoders for this task compared to generative models like GPT-2, as they produce denser and more informative embeddings for graph tasks.

Conclusion

LATEX-GCL demonstrates that natural language is a superior medium for data augmentation in Text-Attributed Graphs. By utilizing LLMs to perform semantic modifications, the framework avoids the information loss inherent in traditional GCL. While computational costs for LLM calls remain a consideration for massive graphs, the significant performance gains and robustness of LATEX-GCL pave the way for more sophisticated self-supervised learning on complex, text-rich graph data.

Summary updated

Abstract

Graph Contrastive Learning (GCL) is a potent paradigm for self-supervised graph learning that has attracted attention across various application scenarios. However, GCL for learning on Text-Attributed Graphs (TAGs) has yet to be explored. Because conventional augmentation techniques like feature embedding masking cannot directly process textual attributes on TAGs. A naive strategy for applying GCL to TAGs is to encode the textual attributes into feature embeddings via a language model and then feed the embeddings into the following GCL module for processing. Such a strategy faces three key challenges: I) failure to avoid information loss, II) semantic loss during the text encoding phase, and III) implicit augmentation constraints that lead to uncontrollable and incomprehensible results. In this paper, we propose a novel GCL framework named LATEX-GCL to utilize Large Language Models (LLMs) to produce textual augmentations and LLMs' powerful natural language processing (NLP) abilities to address the three limitations aforementioned to pave the way for applying GCL to TAG tasks. Extensive experiments on four high-quality TAG datasets illustrate the superiority of the proposed LATEX-GCL method. The source codes and datasets are released to ease the reproducibility, which can be accessed via this link: https://anonymous.4open.science/r/LATEX-GCL-0712.
BibTeX
@misc{yang2024latex,
  author = {Haoran Yang and Xiangyu Zhao and Sirui Huang and Qing Li and Guandong Xu},
  title = {{LATEX-GCL: Large Language Models (LLMs)-Based Data Augmentation for Text-Attributed Graph Contrastive Learning}},
  year = {2024},
  eprint = {2409.01145},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.01145},
}
Figure