MaxEnt Loss: Calibrating Graph Neural Networks under Out-of-Distribution Shift (Student Abstract)

Dexter NeoAAAI 2024 · DOI

Introduction

Modern Graph Neural Networks (GNNs) often suffer from a "confidence problem": their predicted probabilities frequently fail to reflect their actual predictive accuracy. This lack of calibration is particularly problematic in safety-critical applications where reliable uncertainty estimates are essential. To address this, we introduce MaxEnt Loss, a novel loss function that leverages the Principle of Maximum Entropy to improve GNN calibration, especially under out-of-distribution (OOD) shifts.

The Calibration Challenge in GNNs

Calibration ensures that if a model predicts an outcome with 90% confidence, it is correct 90% of the time. While many neural networks are poorly calibrated, GNNs are often "underconfident," meaning they are more accurate than their probability scores suggest.

This issue is exacerbated by Out-of-Distribution (OOD) shifts. While image-based models face shifts like lighting changes or blur, graphs encounter unique shifts such as changes in node resolution. Training on high-resolution graphs and testing on low-resolution ones typically causes calibration to degrade significantly.

Comparison of distribution shifts in images (transformations) versus graphs (resolution shifts).

Methodology: MaxEnt Loss

MaxEnt Loss builds upon Focal Loss, a common alternative to Cross-Entropy that emphasizes "hard" examples. The core innovation is the integration of a Mean Constraint based on the Principle of Maximum Entropy.

The loss function is defined as: $$\mathcal{L}_{MaxEnt} = \mathcal{L}_F + \lambda_1 \left(\sum_k y p_k - \mu\right)$$

Where:

  1. $\mathcal{L}_F$ (Focal Loss): Handles the base classification task.
  2. $\mu$ (Global Constraint): Calculated from the prior class distribution in the data.
  3. $\lambda_1$ (Lagrange Multiplier): Solved numerically using the Newton-Raphson method.

By penalizing deviations from expected class ratios, this constraint forces the model to remain "grounded," leading to better-calibrated probability distributions.

Experimental Setup

We evaluated MaxEnt Loss using a graph-based version of the Celeb-A dataset. Facial images were converted into graphs where nodes represent facial landmarks and edges are formed via triangulation. The primary task was multi-label facial recognition.

Graph representation of facial landmarks used for multi-label recognition tasks.

To test OOD robustness, we introduced resolution shifts, varying the number of nodes and edges between the training and testing sets. This provides a clear framework for assessing how GNNs handle structural distribution changes.

Results and Discussion

Experiments compared standard Cross-Entropy (CE), Focal Loss, and MaxEnt Loss using the Expected Calibration Error (ECE) and Mean Average Precision (MAP):

  • Predictive Power: All loss functions achieved similar MAP, indicating that raw predictive performance remained stable.
  • Calibration Improvements: CE loss showed the poorest calibration. Focal loss improved ECE by approximately 10%, while MaxEnt Loss provided an additional 5% improvement over Focal Loss.
  • OOD Robustness: MaxEnt Loss proved the most robust under resolution shifts, maintaining more reliable confidence scores than the baselines despite the performance dip common to all models under OOD conditions.

The Principle of Maximum Entropy acts as a sophisticated regularizer, preventing the model from becoming biased in its confidence scores by tying them to known class priors.

Conclusion and Future Work

MaxEnt Loss significantly improves GNN calibration without sacrificing predictive accuracy. Future research will expand this work by testing the loss function across a broader range of GNN architectures (beyond GCNConv) and more diverse benchmarks, such as the GOOD (Graph Out-of-Distribution) dataset, covering domains like chemistry and social networks.

Summary updated

BibTeX
@inproceedings{neo2024maxent,
  author = {Dexter Neo},
  title = {{MaxEnt Loss: Calibrating Graph Neural Networks under Out-of-Distribution Shift (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30487},
}
Figure