Abstract
Curriculum learning provides a systematic approach to training. It refines training progressively, tailors training to task requirements, and improves generalization through exposure to diverse examples. We present a curriculum learning approach that builds on existing knowledge about text and graph complexity formalisms for training with text graph data. The core part of our approach is a novel data scheduler, which employs “spaced repetition” and complexity formalisms to guide the training process. We demonstrate the effectiveness of the proposed approach on several text graph tasks and graph neural network architectures. The proposed model gains more and uses less data; consistently prefers text over graph complexity indices throughout training, while the best curricula derived from text and graph complexity indices are equally effective; and it learns transferable curricula across GNN models and datasets. In addition, we find that both node-level (local) and graph-level (global) graph complexity indices, as well as shallow and traditional text complexity indices play a crucial role in effective curriculum learning.
Introduction
Training Graph Neural Networks (GNNs) on text graphs—where nodes represent entities with rich textual descriptions, such as citation networks or biomedical databases—presents unique challenges. Standard training paradigms typically treat all data points as equally difficult from the outset. However, nodes in these graphs exhibit varying levels of complexity, both topologically (e.g., edge-case cluster positions) and linguistically (e.g., technical jargon density).
Existing Curriculum Learning (CL) methods often focus on either structural graph properties or linguistic features in isolation. Moreover, most CL schedulers are monotonic, progressively adding data without strategically revisiting or delaying specific samples. This paper introduces TGCL (Complexity-Guided Curriculum Learning), a framework that leverages established formalisms of both text and graph complexity to guide GNN training more efficiently.
The TGCL Framework
The TGCL framework is built upon three primary components designed to optimize the learning trajectory:
- Multiview Complexity Formalisms: The system evaluates data difficulty through a comprehensive set of 26 graph indices (e.g., node degree, centrality) and 14 linguistic indices (e.g., SMOG readability, sentence length).
- Flexible Competence Function: This function manages the "budget" of training samples available at any given epoch. A tunable parameter ($\alpha$) controls the ramp-up speed from easy to hard samples.
- Spaced Repetition Scheduler: Inspired by cognitive science, this scheduler calculates a delay parameter for each complexity index. If the model masters samples ranked by a specific index, the scheduler delays their re-introduction, allowing the GNN to prioritize harder, unlearned data.
Experimental Evaluation and Insights
The researchers evaluated TGCL across five real-world datasets, including citation networks (Ogbn-arxiv, Cora, Citeseer) and biomedical relations (GDPR, PGR). The framework was tested across four GNN architectures: GraphSAGE, GCN, GAT, and GTNN.
Key Results
- Performance and Efficiency: TGCL outperformed state-of-the-art baselines, achieving an average improvement of 5.1 absolute points in accuracy/F1 score. Notably, for node classification tasks, the model utilized 39.2% less training data on average compared to standard "No-CL" training.
- Training Dynamics: As shown in Figure 6, the TGCL scheduler dynamically adjusts the number of training examples, often using significantly fewer samples than the full dataset (No-CL) while maintaining or exceeding performance.
- Linguistic Preference: A surprising finding was that the model consistently prioritized linguistic complexity indices over structural graph indices, suggesting that text provides a more reliable signal for difficulty in these tasks.
- Transferability: Curricula learned on one dataset or architecture proved transferable to others, indicating that "difficulty" in text graphs possesses universal characteristics.
Conclusion
TGCL demonstrates that "better-timed data" is often superior to "more data." By strategically delaying mastered samples and utilizing a multi-view approach to complexity, the framework achieves higher performance with reduced computational load. While the pre-processing of complexity indices introduces some overhead, the gains in data efficiency and model robustness make it a compelling approach for complex text-graph domains.
Summary updated