Fairness-enhancing classification methods for non-binary sensitive features - How to fairly detect leakages in water distribution systems

Janine Strotherm, Inaam Ashraf, Barbara HammerPeerJ Comput. Sci 2024 · DOI

TL;DR

This paper tackles the critical challenge of ensuring fairness in AI systems used for water distribution systems (WDSs), specifically for leakage detection. They propose novel, generalized fairness definitions applicable to non-binary sensitive features and demonstrate that standard leakage detection methods are unfair. Their proposed fairness-enhancing framework successfully mitigates this unfairness while maintaining performance, offering a practical solution for trustworthy AI in critical infrastructure.

What Problem Does This Paper Address?

As AI systems become increasingly integrated into critical infrastructure like water distribution systems (WDSs), ensuring their fairness is paramount. Decisions made by these systems can disproportionately affect different groups of consumers, leading to inequitable service. This paper focuses on leakage detection in WDSs, a socioeconomically relevant application. Traditional AI models for leakage detection, while effective, may exhibit bias due to how they treat different geographical areas or consumer groups within the network. The challenge lies in defining and implementing fairness for these systems, especially when sensitive attributes (like location within the network) are not simple binary categories. The authors aim to bridge this gap by developing generalized fairness definitions and practical methods to make WDS leakage detection fairer.

Methods

The core of this paper lies in its novel approach to fairness in machine learning, particularly for non-binary sensitive features.

  • Generalized Fairness Definitions: They extend existing group fairness notions (like disparate impact and demographic parity) to handle multiple, non-binary sensitive features. This is achieved by building upon the mathematical concept of conditional independence and deriving easy-to-test criteria.
  • Leakage Detection Pipeline: They formalize a standard leakage detection pipeline that involves:
  • Virtual Sensors: Training regression models to predict pressure at various points in the WDS based on sensor data.
  • Pressure Residuals: Calculating the difference between actual and predicted pressures.
  • Leakage Detector: A classifier that uses these residuals to predict leakage.
  • Fairness-Enhancing Framework: They propose an "in-process" method that integrates fairness constraints directly into the training of the leakage detection model. This involves reformulating the problem as an optimization problem where both performance (e.g., accuracy) and fairness metrics (e.g., disparate impact) are optimized, often using barrier or penalty functions.
  • Instantiations: They present specific implementations of their framework, including methods that optimize for fairness under performance constraints and vice-versa, using different loss functions and algorithmic choices.

Datasets

The authors evaluate their methods on two WDS datasets:

  • Hanoi WDS: A smaller, simulated WDS with 32 nodes and 3 sensors. Data was generated for different leakage scenarios and sizes (5, 10, 15 cm diameter).
  • L-Town WDS: A larger, more realistic simulated WDS with 785 nodes and 33 sensors. Data was generated for different leakage scenarios and sizes (1.9, 2.3, 2.7 cm diameter).

The paper notes that real-world data for such systems is limited, necessitating simulation for evaluation.

Key Insights and Ideas

  • Non-binary fairness is crucial: Standard fairness definitions, often designed for binary sensitive features (like gender), are insufficient for complex real-world scenarios like WDSs where location or group affiliation can be multi-faceted.
  • Leakage detection can be unfair: Even seemingly neutral leakage detection models can exhibit disparate impact, meaning they perform differently across various protected groups within the WDS.
  • Fairness and performance are interdependent: There's a trade-off between achieving high fairness and maintaining optimal model performance (accuracy). However, this trade-off is not always linear and can be managed through hyperparameter tuning.
  • Generalized definitions work: The proposed generalized fairness definitions and the resulting fairness-enhancing methods are effective in practice, demonstrating improvements in fairness metrics.
  • Hyperparameters are key to trade-offs: The choice of hyperparameters in the fairness-enhancing methods allows policymakers to explicitly control the balance between fairness and performance, though this requires careful tuning.

Key Results

  • Standard methods are unfair: The paper demonstrates that common leakage detection methods exhibit unfairness, particularly in terms of disparate impact. For instance, in the Hanoi WDS, DI scores were often below the desirable threshold of 0.8.
  • Fairness-enhancing methods improve fairness: The proposed methods, especially the DI+ACC-ndb-max method, significantly improved fairness scores (e.g., achieving DI scores closer to 1.0) compared to baseline methods.
  • Performance trade-off: While improving fairness, there was a slight decrease in accuracy, typically around 0.03-0.06 points below the optimal accuracy of baseline methods. However, the increase in fairness was often larger than the decrease in accuracy.
  • Equivalence of fairness metrics: In the specific context of their WDS application, the paper empirically confirmed the theoretical equivalence between disparate impact and equal opportunity scores, simplifying the evaluation process.
  • Scalability: The DI+ACC-ndb-max method showed promising results on both the smaller Hanoi WDS and the more complex L-Town WDS, suggesting its potential for real-world applications.

Limitations and Future Work

  • Non-convexity and Scalability: The authors acknowledge that the optimization problems involved in their fairness-enhancing methods are non-convex, which can lead to solutions that depend on initial parameters and may not be globally optimal. This also impacts training time, especially for larger networks. Future work could investigate methods to address this.
  • Data Availability: The reliance on simulated data due to the scarcity of real-world WDS datasets is a limitation. Future work could focus on obtaining and utilizing more real-world data.
  • Broader Applications: While leakage detection is a key focus, the authors suggest that their fairness framework can be adapted to other ML-based services in WDSs, such as contamination detection. Further research in these areas is warranted.
  • Legal vs. Practical Fairness: The paper highlights that while their methods can satisfy legal notions of disparate treatment and impact, the practical implications of the fairness-performance trade-off ultimately rest with policymakers. Further exploration of how to best navigate this trade-off in different WDS contexts would be valuable.

Summary updated

BibTeX
@article{strotherm2024fairness,
  author = {Janine Strotherm and Inaam Ashraf and Barbara Hammer},
  title = {{Fairness-enhancing classification methods for non-binary sensitive features - How to fairly detect leakages in water distribution systems}},
  journal = {PeerJ Comput. Sci.},
  year = {2024},
  doi = {10.7717/peerj-cs.2317},
}
Figure