Evaluating graph fairness in transductive learning

Fernanda Lenita Ribeiro, Valentina Shumovskaia, Thomas Davies, Ira KtenaMIDL Short Papers 2022 · OpenReview

TL;DR

This paper investigates how using demographic information (like sex) and data source (like imaging site) in graph-based machine learning models for predicting autism spectrum disorder (ASD) can inadvertently lead to unfairness, where the model performs worse for certain subgroups. They found that the structure of the graph used to represent this information, rather than how the data is split for training, has a significant impact on fairness.

What Problem Does This Paper Address?

Machine learning models are increasingly used in healthcare for tasks like diagnosing diseases. When these models use data from large, multi-site studies, they often incorporate non-imaging information like participant demographics (e.g., sex) or data acquisition site. This information can be encoded into "population graphs" to help the model learn better. However, real-world datasets are often imbalanced, meaning certain groups (e.g., males vs. females, or participants from specific sites) might be underrepresented or have different disease prevalence. This imbalance can lead to models that perform well overall but are unfair, exhibiting significantly worse prediction accuracy for certain subgroups. This paper specifically addresses the question of whether and how using demographic and site information within population graphs for ASD prediction can exacerbate these fairness issues, and how different ways of structuring these graphs and splitting the data affect this fairness.

Methods

The authors employ a semi-supervised learning approach using Graph Convolutional Networks (GCNs) to predict ASD diagnosis. The core of their method involves constructing "phenotypic population graphs." These graphs represent relationships between individuals, where the edges are weighted based on both their brain connectivity (connectome similarity) and phenotypic features. They explore four different graph structures:

  1. SexGraph: Graph weighted by participant sex.
  2. SiteGraph: Graph weighted by acquisition site.
  3. SexSiteGraph: Graph weighted by both sex and acquisition site.
  4. CompleteGraph: A baseline graph that doesn't leverage phenotypic information (effectively, a fully connected graph).

To evaluate fairness, they focus on the True Positive Rate (TPR) gap between males and females, which is a key fairness metric (part of "equality of odds"). They also investigate four different stratification strategies for splitting the data into training and testing sets:

  1. Stratification by diagnosis only.
  2. Stratification by diagnosis and sex.
  3. Stratification by diagnosis and acquisition site.
  4. Stratification by sex and acquisition site.

This allows them to disentangle the effects of data splitting versus graph structure on fairness.

Datasets

The study uses the ABIDE database, a large-scale dataset comprising functional neuroimaging and phenotypic data from 871 participants (403 neurodiverse, 468 neurotypical) across multiple international acquisition sites. This dataset is noted for its significant variation in the number of participants per site, making it a challenging testbed for exploring biases.

Key Insights and Ideas

  • Graph structure matters more than data splitting for fairness: The most striking finding is that the way the population graph is constructed (i.e., which phenotypic information is encoded and how) has a more significant impact on fairness than the strategy used to split the data into training and testing sets.
  • Incorporating sensitive attributes can introduce bias: While using demographic information like sex in the graph can improve overall performance, it can also lead to a larger TPR gap between males and females, indicating poorer performance for one group.
  • Site information's impact is nuanced: The effect of including acquisition site information in the graph on fairness is less straightforward and depends on how it interacts with other attributes.
  • Fairness metrics are crucial: Relying solely on overall performance metrics (like accuracy) can mask significant disparities in performance across subgroups.

Key Results

Figure 1 shows the absolute difference in true positive rates between males and females across different stratifications and graph structures.

  • The SexGraph and SexSiteGraph structures consistently show larger TPR differences (indicating less fairness) compared to the CompleteGraph and SiteGraph.
  • The CompleteGraph (no phenotypic information) and SiteGraph (only acquisition site) exhibit the most favorable fairness metrics, with smaller TPR gaps.
  • The stratification strategies had a less pronounced effect on the TPR differences compared to the graph structures.

Limitations and Future Work

The authors acknowledge that while they explored several graph structures and stratification strategies, a comprehensive analysis of all possible combinations and fairness metrics is beyond the scope of this paper. Future work could involve developing novel graph construction methods that explicitly optimize for fairness alongside predictive performance, and exploring other sensitive attributes beyond sex and acquisition site. They also suggest investigating the underlying reasons for the observed performance disparities across subgroups.

Summary updated

Abstract

Recent work on neuroimaging has demonstrated significant benefits of using population graphs to capture non-imaging information in the prediction of neurodegenerative and neurodevelopmental disorders. These non-imaging attributes may not only contain demographic information about the individuals, e.g. age or sex, but also the acquisition site, as imaging protocols might significantly differ across sites in large-scale studies. In addition, recent studies have highlighted the need to investigate potential biases in the classifiers devised using large-scale datasets, which might be imbalanced in terms of one or more sensitive attributes. This can be exacerbated when employing these attributes in a population graph to explicitly introduce inductive biases to the machine learning model and lead to disparate predictive performance across sub-populations. In this work, we explore the impact of stratification strategies and graph structures on the fairness of a semi-supervised classifier that relies on a population graph for the prediction of autism-spectrum disorder.
BibTeX
@inproceedings{ribeiro2022evaluating,
  author = {Fernanda Lenita Ribeiro and Valentina Shumovskaia and Thomas Davies and Ira Ktena},
  title = {{Evaluating graph fairness in transductive learning}},
  booktitle = {MIDL Short Papers},
  year = {2022},
  url = {https://openreview.net/forum?id=RojT1Dh9bzE},
}
Figure