Argumentative Debates for Transparent Bias Detection [Technical Report]

Hamed Ayoobi, Nico Potyka, Anna Rapberger, Francesca ToniarXiv 2025 · DOI · arXiv

Introduction

As artificial intelligence systems are increasingly deployed in high-stakes domains such as healthcare, finance, and criminal justice, the need for robust bias detection has become paramount. However, existing fairness tools often operate as "black boxes," providing numerical scores without explaining the underlying reasons for a model's biased behavior. This lack of transparency hinders trust and complicates mitigation efforts.

To address this, we present ABIDE (Argumentative BIas detection by DEbate), a framework that transforms bias detection into a structured, human-readable debate. By leveraging formal argumentation theory, ABIDE provides faithful justifications for its findings, moving beyond global statistics to uncover local biases within specific sub-groups.

Methodology: Bias Detection as Argumentation

ABIDE structures the detection process as a Quantitative Bipolar Argumentation Framework (QBAF). The system identifies "neighborhoods" of similar individuals and evaluates whether protected groups are disadvantaged relative to their peers.

The ABIDE framework uses a QBAF structure where bias arguments are supported by success rates and attacked by critical questions regarding data quality.

The core innovation lies in the use of Critical Questions (CQs). Rather than blindly accepting a bias claim, the framework subjects it to "attacks" based on data quality metrics:

  • Significance: Is the neighborhood large enough to provide a statistically meaningful comparison?
  • Diversity: Does the neighborhood contain a sufficient mix of individuals to ensure a fair baseline?
  • Objectivity: Is the neighborhood mathematically well-defined (e.g., convex)?

The final output is a numerical strength value for the presence of bias, derived from the interaction of these supporting and attacking arguments.

Transparent Justification through Debate

A key feature of ABIDE is its ability to present these complex logical structures as a natural language dialogue. This allows auditors to follow the reasoning process step-by-step, ensuring that the explanation is faithful to the underlying model logic.

An example of a structured debate generated by ABIDE, illustrating how the system handles counter-arguments regarding neighborhood size and diversity.

By presenting the detection results as a debate, the final decision is not a post-hoc interpretation but a direct result of the arguments weighed during the process. This structure naturally handles statistical uncertainty without needing complex hybrid probabilistic models.

Experimental Results

The framework was evaluated using synthetic models with known biases and real-world datasets, including COMPAS and the Adult Census Income (ACI) dataset.

  • Enhanced Sensitivity: On the COMPAS recidivism dataset, ABIDE identified 77 instances of bias against African-American individuals, a significant improvement over the baseline IRB method, which detected only 2.
  • LLM Auditing: When applied to ChatGPT-4o, ABIDE uncovered 129 instances of racial bias and 6 instances of gender bias that were previously undetected by baseline methods.
  • Efficiency and Robustness: ABIDE maintained high precision and recall across varying neighborhood sizes ($K=50$ to $200$) and demonstrated runtimes nearly ten times faster than previous argumentative baselines (e.g., ~4 seconds vs ~48 seconds for certain tests).

Conclusion and Future Directions

ABIDE represents a significant step toward transparent and interpretable AI auditing. By framing bias detection as a debate, it provides stakeholders with the "why" behind fairness metrics. Future work will explore human-in-the-loop scenarios, allowing auditors to interactively challenge the system's arguments, and expanding the framework to support multi-agent consensus building.

Summary updated

Abstract

As the use of AI in society grows, addressing emerging biases is essential to prevent systematic discrimination. Several bias detection methods have been proposed, but, with few exceptions, these tend to ignore transparency. Instead, interpretability and explainability are core requirements for algorithmic fairness, even more so than for other algorithmic solutions, given the human-oriented nature of fairness. We present ABIDE (Argumentative BIas detection by DEbate), a novel framework that structures bias detection transparently as debate, guided by an underlying argument graph as understood in (formal and computational) argumentation. The arguments are about the success chances of groups in local neighbourhoods and the significance of these neighbourhoods. We evaluate ABIDE experimentally and demonstrate its strengths in performance against an argumentative baseline.
BibTeX
@misc{ayoobi2025argumentative,
  author = {Hamed Ayoobi and Nico Potyka and Anna Rapberger and Francesca Toni},
  title = {{Argumentative Debates for Transparent Bias Detection [Technical Report]}},
  year = {2025},
  eprint = {2508.04511},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.04511},
}
Figure