Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems

Richard Tomsett, Dave Braines, Dan Harborne, Alun D. Preece, Supriyo ChakrabortyarXiv 2018 · arXiv

Abstract

Interpretability is frequently cited as a requirement for trustworthy machine learning, yet it remains a poorly defined concept often treated as a static property of a model. This paper argues that interpretability is inherently relational, existing between a system and a specific agent. We propose a "role-based ecosystem model" that identifies six distinct stakeholder roles: Creators, Operators, Executors, Decision-subjects, Data-subjects, and Examiners. By categorizing these agents and their specific goals, the framework provides a structured approach to defining, measuring, and delivering appropriate explanations. This model moves the field toward a more nuanced, human-centered understanding of how AI systems communicate their logic to diverse audiences.

Introduction: The Problem of One-Size-Fits-All Interpretability

In the rush to make AI "explainable," the community has struggled with a fundamental question: Interpretable to whom? Currently, interpretability is often treated as a vague, universal metric. However, the information required by a developer debugging a neural network is vastly different from that needed by a patient receiving a medical diagnosis or a regulator auditing a bank for systemic bias.

The authors argue that without a formal way to identify who the "user" is and what their goals are, we cannot effectively measure or improve interpretability. They aim to move the conversation from "Is this model interpretable?" to "Does this system provide the specific information this agent needs to achieve their goals?"

The Role-Based Ecosystem Model

The authors introduce a conceptual framework to categorize the agents interacting with or affected by an ML system. Central to this framework is the distinction between five key terms:

  • Explanation: The specific information provided by the system regarding a decision.
  • Interpretation: The internal mental model or understanding an agent forms based on that explanation.
  • Explainability: The technical capacity of the system to generate these clarifications.
  • Transparency: The system's ability to reveal its internal logic, parameters, or training data.
  • Interpretability: The degree to which an agent can successfully utilize explanations and transparency to gain a functional understanding.

The core of the paper is the definition of six primary agent roles:

  1. Creators: Those who build and own the system (e.g., data scientists, software engineers).
  2. Operators: Those who directly interact with the system (e.g., a clinician entering patient data).
  3. Executors: Those who make decisions based on the system's output (e.g., a senior doctor).
  4. Decision-subjects: Those affected by the executor's decision (e.g., a loan applicant).
  5. Data-subjects: Those whose personal data was used to train the model.
  6. Examiners: Auditors or investigators (e.g., regulatory bodies or safety testers).
The machine learning ecosystem model illustrating the relationships between the system and various stakeholder roles.

Key Insights: Interpretability as a Relational Property

The ecosystem model highlights that interpretability is not a fixed attribute but a bridge between system-side capabilities (Explainability/Transparency) and agent-side cognition (Interpretation). This perspective reveals several critical dynamics:

  • Goal Mismatch: Stakeholders often have conflicting objectives. For instance, a Decision-subject might seek high interpretability to "game" a scoring system, while Creators may intentionally limit transparency to protect intellectual property or prevent system manipulation.
  • Transparency vs. Explainability: These needs are distinct. A Data-subject may prioritize transparency regarding data usage and privacy, whereas an Executor requires actionable explainability to justify a specific recommendation and assume professional responsibility.
  • The Examiner's Forensic Requirement: Auditors face unique challenges, such as the need to recreate past system states. In dynamic environments like reinforcement learning, providing a forensic explanation for a historical decision remains a significant technical hurdle.

Applications and Real-World Scenarios

The paper maps these roles onto complex domains to demonstrate the model's utility:

  • Healthcare: In a diagnostic setting, the Operator (doctor) needs to verify input accuracy, while the Decision-subject (patient) requires an explanation that facilitates informed consent.
  • Defense: A mission commander (Executor) requires explanations centered on mission success and risk, while a tribunal (Examiner) requires a forensic trail to evaluate legal and ethical compliance.

Limitations and Future Directions

While the role-based model provides a necessary taxonomy, it is a foundational step. Current limitations and areas for future work include:

  • Role Overlap: In many consumer applications, a single individual may occupy multiple roles (e.g., a smartphone user is the operator, executor, and decision-subject). The model needs to account for these collapsed roles.
  • Quantification: The framework is currently qualitative; future work is needed to develop mathematical metrics for interpretability tailored to each role.
  • Automated Governance: The model could eventually inform automated privacy filters that dynamically adjust explanation depth based on an agent's verified role and authorization level.

Summary updated

Abstract

Several researchers have argued that a machine learning system's interpretability should be defined in relation to a specific agent or task: we should not ask if the system is interpretable, but to whom is it interpretable. We describe a model intended to help answer this question, by identifying different roles that agents can fulfill in relation to the machine learning system. We illustrate the use of our model in a variety of scenarios, exploring how an agent's role influences its goals, and the implications for defining interpretability. Finally, we make suggestions for how our model could be useful to interpretability researchers, system developers, and regulatory bodies auditing machine learning systems.
BibTeX
@misc{tomsett2018interpretable,
  author = {Richard Tomsett and Dave Braines and Dan Harborne and Alun D. Preece and Supriyo Chakraborty},
  title = {{Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems}},
  year = {2018},
  eprint = {1806.07552},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1806.07552},
}
Figure