How do loss functions impact the performance of graph neural networks?

Gabriel Jonas Duarte, Tamara Arruda Pereira, Erik Jhones Freitas do Nascimento, Diego Mesquita, Amauri H. Souza2021 · DOI

Abstract: Graph neural networks (GNNs) have become the de facto approach for supervised learning on graph data. To train these networks, most practitioners employ the categorical cross-entropy (CE) loss, largely due to its probabilistic interpretability as the negative log of the categorical/softmax likelihood. Nonetheless, recent works in other domains have shown that deep learning models can benefit from alternative loss functions; for instance, symmetric losses like mean absolute error (MAE) can improve robustness to label noise. In this work, we gauge the impact of different loss functions on GNN performance for node classification under conditions of noisy labels and varying sample sizes. In contrast to findings in Euclidean domains, our results show no significant difference between models trained with CE and other classical loss functions in these scenarios.

Introduction

In deep learning, Categorical Cross-Entropy (CE) is the standard loss function for classification tasks. However, in traditional Euclidean domains like computer vision, CE is known to be sensitive to label noise. Researchers have found that "symmetric" losses, such as Mean Absolute Error (MAE), can significantly improve robustness in these settings.

Despite the rapid advancement of Graph Neural Networks (GNNs), the impact of different loss functions on graph-structured data remains largely unexplored. This study investigates whether the robustness benefits of alternative losses translate to GNNs for node classification when labels are noisy or scarce.

Methodology

The researchers conducted an empirical study comparing four distinct loss functions:

  1. Cross-Entropy (CE): The standard probabilistic approach.
  2. Mean Squared Error (MSE): Adapted for classification tasks.
  3. Mean Absolute Error (MAE): Known for noise robustness in non-graph tasks.
  4. Weston-Watkins Hinge Loss: A multi-class SVM-based loss.

These were tested using three common GNN architectures—GCN, SGC, and GAT—across five benchmark datasets: Cora, Citeseer, Pubmed, DBLP, and Wiki-CS. The experiments focused on two primary challenges: symmetric label noise (randomly flipping training labels) and data scarcity (training with very few labeled nodes per class).

Results and Discussion

Robustness to Label Noise

A key finding in computer vision is that MAE is significantly more robust to noisy labels than CE. Surprisingly, this "robustness gap" does not appear to exist for GNNs. As shown in Figure 1, while accuracy decreases as the noise level increases, all loss functions perform remarkably similarly across different architectures and datasets.

Accuracy of various loss functions under increasing symmetric label noise across GNN architectures and datasets.

The results suggest that the specific mathematical form of the loss function is less critical in the graph domain. One hypothesis is that the neighborhood aggregation (message-passing) mechanism in GNNs acts as a natural regularizer. By averaging features from a node's neighbors, the model may inherently smooth out the impact of individual noisy labels before the loss is even calculated.

Performance under Data Scarcity

The study also examined how these loss functions scale with limited training data. As the number of labeled nodes per class increases from 5 to 60, the performance curves for CE, MSE, MAE, and Hinge loss are nearly indistinguishable.

GNN performance across different loss functions as the number of labeled nodes per class increases, showing nearly identical scaling.

This identical scaling further reinforces the conclusion that, for semi-supervised node classification, the choice of loss function is not a primary driver of performance. Even in extreme data-scarce scenarios (e.g., 5 nodes per class), no single loss function consistently provides a significant advantage.

Conclusion

This empirical study reveals a surprising twist: the choice of loss function—a critical decision in other deep learning fields—has a negligible impact on GNN performance for node classification. Whether facing high label noise or extreme data scarcity, standard Cross-Entropy performs as well as specialized "robust" losses like MAE. Future work may explore whether these findings hold for other graph tasks, such as link prediction, or if alternative losses impact model calibration even when accuracy remains constant.

Summary updated

Abstract

Graph neural networks (GNNs) have become the de facto approach for supervised learning on graph data.To train these networks, most practitioners employ the categorical cross-entropy (CE) loss. We can attribute this largely to the probabilistic interpretability of models trained using CE, since it corresponds to the negative log of the categorical/softmax likelihood.We can attribute this largely to the probabilistic interpretation of CE, since it corresponds to the negative log of the categorical/softmax likelihood.Nonetheless, recent works have shown that deep learning models can benefit from adopting other loss functions. For instance, neural networks trained with symmetric losses (e.g., mean absolute error) are robust to label noise. Nonetheless, loss functions are a modeling choice and other training criteria can be employed — e.g., hinge loss and mean absolute error (MAE). Perhaps surprisingly, the effect of using different losses on GNNs has not been explored. In this preliminary work, we gauge the impact of different loss functions to the performance of GNNs for node classification under i) noisy labels and ii) different sample sizes. In contrast to findings on Euclidean domains, our results for GNNs show that there is no significant difference between models trained with CE and other classical loss functions on both aforementioned scenarios.
BibTeX
@misc{duarte2021how,
  author = {Gabriel Jonas Duarte and Tamara Arruda Pereira and Erik Jhones Freitas do Nascimento and Diego Mesquita and Amauri H. Souza},
  title = {{How do loss functions impact the performance of graph neural networks?}},
  year = {2021},
  doi = {10.21528/cbic2021-161},
}

Loading related content...

Figure