A Model for Accountable Ordinal Sorting

Khaled Belahcène, Christophe Labreuche, Nicolas Maudet, Vincent Mousseau, Wassila Ouerdane

Introduction: The Accountability Gap

In the era of the GDPR and the "right to explanation," a critical gap has emerged between algorithmic transparency and accountability. While transparency allows us to see the code or weights behind a decision, accountability requires that a human decision-maker can "own" and defend a recommendation to stakeholders.

Most Multi-Criteria Decision Analysis (MCDA) tools rely on "black-box" artifacts like numerical weights or scoring functions. Explaining a job candidate's rejection based on a weighted average is often unpersuasive. This paper introduces a multicriteria ordinal sorting model designed for accountability, allowing for human-readable arguments that mirror natural reasoning.

The Proposed Model: Non-Compensatory Ordinal Sorting

The authors propose a model built on four core design principles to ensure accountability:

  1. Indirect Elicitation: The system learns from "reference objects"—examples already categorized by the user—rather than asking for abstract weights.
  2. Robustness: Recommendations are valid across all possible parameters consistent with the user's examples.
  3. Non-Compensation: A high performance in one area cannot mask a failure in another. This mimics strict human requirements (e.g., a car is not "Safe" if it lacks airbags, regardless of its braking performance).
  4. No Frontiers: The model avoids arbitrary numerical thresholds, relying instead on direct comparisons to reference objects.

The technical core involves identifying Sufficient Coalitions of criteria. If a user ranks Object A higher than Object B because it performs better on criteria {speed, cost}, the system identifies this subset as "sufficient" to justify the ranking.

Illustrative Dataset

To demonstrate the model, the authors use a synthetic dataset with four criteria ($a, b, c, d$) and six reference objects ($A_1, A_2, B_1, B_2, C_1, C_2$) pre-assigned to three ordered classes (\star, \star\star, \star\star\star).

Table 1: Synthetic illustrative dataset showing criteria values and class assignments for reference objects and candidates X and Y.

Two new candidates, $X$ and $Y$, are introduced to show how the model handles sorting and potential "incomparability."

Key Insights: From Trust to Accountability

A central argument of the paper is that while trust relies on accuracy, accountability relies on persuasion. A model is accountable only if its logic can be transferred from the machine to the human.

The model utilizes the "power of impossibility," focusing on proving why an object cannot belong to certain classes. This "burden of proof" approach makes the logic sturdier. Furthermore, by treating criteria as purely ordinal (better vs. worse) rather than cardinal (numerical values), the model avoids the jargon of numerical scaling that often confuses non-experts.

Results and Argumentation

The paper formalizes several argument schemes, such as OUTRANKING and WEAK DOMINANCE, which can be automatically populated to explain a decision.

Table 2: Analysis of sufficient coalitions and comparisons between reference objects and candidates X and Y.

As shown in the analysis above, the model doesn't always provide a single class; it may provide an interval of possible classes. For instance, it might conclude a candidate is "at least \star\star but no higher than \star\star\star." It also identifies conflicts, such as when a user's initial examples are inconsistent, providing a vital feedback loop for decision-makers.

Limitations and Future Work

Current limitations include the handling of inconsistent preference information and computational complexity as the number of criteria grows. Future work aims to move toward "dialogue games" where users and systems can debate the importance of specific criteria, further enhancing the accountability of the final decision.

Summary updated

Abstract

International audience
BibTeX
@inproceedings{belahcene2017model,
  author = {Khaled Belahcène and Christophe Labreuche and Nicolas Maudet and Vincent Mousseau and Wassila Ouerdane},
  title = {{A Model for Accountable Ordinal Sorting}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/113},
}
Figure