Abstract: Graph neural networks (GNNs) have become the de facto approach for supervised learning on graph data. To train these networks, most practitioners employ the categorical cross-entropy (CE) loss, largely due to its probabilistic interpretability as the negative log of the categorical/softmax likelihood. Nonetheless, recent works in other domains have shown that deep learning models can benefit from alternative loss functions; for instance, symmetric losses like mean absolute error (MAE) can improve robustness to label noise. In this work, we gauge the impact of different loss functions on GNN performance for node classification under conditions of noisy labels and varying sample sizes. In contrast to findings in Euclidean domains, our results show no significant difference between models trained with CE and other classical loss functions in these scenarios.
Introduction
In deep learning, Categorical Cross-Entropy (CE) is the standard loss function for classification tasks. However, in traditional Euclidean domains like computer vision, CE is known to be sensitive to label noise. Researchers have found that "symmetric" losses, such as Mean Absolute Error (MAE), can significantly improve robustness in these settings.
Despite the rapid advancement of Graph Neural Networks (GNNs), the impact of different loss functions on graph-structured data remains largely unexplored. This study investigates whether the robustness benefits of alternative losses translate to GNNs for node classification when labels are noisy or scarce.
Methodology
The researchers conducted an empirical study comparing four distinct loss functions:
- Cross-Entropy (CE): The standard probabilistic approach.
- Mean Squared Error (MSE): Adapted for classification tasks.
- Mean Absolute Error (MAE): Known for noise robustness in non-graph tasks.
- Weston-Watkins Hinge Loss: A multi-class SVM-based loss.
These were tested using three common GNN architectures—GCN, SGC, and GAT—across five benchmark datasets: Cora, Citeseer, Pubmed, DBLP, and Wiki-CS. The experiments focused on two primary challenges: symmetric label noise (randomly flipping training labels) and data scarcity (training with very few labeled nodes per class).
Results and Discussion
Robustness to Label Noise
A key finding in computer vision is that MAE is significantly more robust to noisy labels than CE. Surprisingly, this "robustness gap" does not appear to exist for GNNs. As shown in Figure 1, while accuracy decreases as the noise level increases, all loss functions perform remarkably similarly across different architectures and datasets.
The results suggest that the specific mathematical form of the loss function is less critical in the graph domain. One hypothesis is that the neighborhood aggregation (message-passing) mechanism in GNNs acts as a natural regularizer. By averaging features from a node's neighbors, the model may inherently smooth out the impact of individual noisy labels before the loss is even calculated.
Performance under Data Scarcity
The study also examined how these loss functions scale with limited training data. As the number of labeled nodes per class increases from 5 to 60, the performance curves for CE, MSE, MAE, and Hinge loss are nearly indistinguishable.
This identical scaling further reinforces the conclusion that, for semi-supervised node classification, the choice of loss function is not a primary driver of performance. Even in extreme data-scarce scenarios (e.g., 5 nodes per class), no single loss function consistently provides a significant advantage.
Conclusion
This empirical study reveals a surprising twist: the choice of loss function—a critical decision in other deep learning fields—has a negligible impact on GNN performance for node classification. Whether facing high label noise or extreme data scarcity, standard Cross-Entropy performs as well as specialized "robust" losses like MAE. Future work may explore whether these findings hold for other graph tasks, such as link prediction, or if alternative losses impact model calibration even when accuracy remains constant.
Summary updated