Fanoos: Multi-Resolution, Multi-Strength, Interactive Explanations for Learned Systems

David Bayani, Stefan Mitsch

Introduction

As machine learning (ML) models are increasingly deployed in safety-critical and financially sensitive domains, their "black box" nature becomes a significant liability. While the field of Explainable AI (XAI) has grown rapidly, most existing tools suffer from two primary flaws: they provide static, "one-size-fits-all" abstractions and often lack formal guarantees. To address these challenges, we introduce Fanoos, an interactive framework that allows users to explore model behavior at varying levels of detail and formal "strength."

Methodology

Fanoos treats explanation generation as a search and refinement process. Its core pipeline utilizes a Counter-Example Guided Abstraction Refinement (CEGAR) approach to represent the model's input/output space using hyper-cubes (boxes). These boxes are propagated through the model to identify regions satisfying specific user queries.

Query Types and Strengths

Fanoos supports a variety of question types, allowing users to probe the model for formal guarantees or statistical tendencies. For "hard" questions requiring high confidence, the system employs SAT solvers (like Z3) to provide provably sound descriptions. For "usual" behavior, it utilizes statistical sampling to avoid rare edge cases that might clutter a formal explanation.

Summary of question types supported by Fanoos, including formal definitions and examples.

Interactive Multi-Resolution Refinement

A key feature of Fanoos is its ability to adjust the resolution of an explanation based on user feedback. Users can request that an explanation be "more abstract" or "less abstract." In response, Fanoos adjusts internal hyperparameters—such as box size or predicate specificity—and re-runs its search. This results in a multi-resolution partitioning of the input space, as shown below.

Multi-resolution partitioning of the input space (freemem vs lread) using hyper-cubes.

Experimental Evaluation

The authors evaluated Fanoos on two distinct systems:

  • Inverted Double Pendulum (IDP): A complex 7-dimensional robotics control policy trained via reinforcement learning (PPO2).
  • CPU Usage (CPU): A 5-dimensional polynomial regression task sourced from the OpenML dataset.

Key Results and Insights

The experiments demonstrated that Fanoos successfully navigates the "abstraction ladder":

  • Structural Trends: Requests for "more abstract" explanations led to shorter descriptions with fewer disjuncts and more general terms. Conversely, "less abstract" requests provided high-granularity detail; in the CPU model, this increased the box count by over 8,000%.
  • Groundedness and Fidelity: Because explanations are built from predicates tied directly to the model's variables, they are grounded in actual logic rather than post-hoc rationalizations. The use of formal verification ensures that these descriptions are mathematically sound over the defined regions.
  • The "Washing Out" Problem: Fanoos employs specific scoring metrics during its predicate selection phase to ensure that granular details are not "washed out" by broader, weaker predicates when a user requests high detail.

Conclusion and Future Work

Fanoos provides a flexible, interactive bridge between high-level human intuition and low-level formal guarantees. While currently reliant on pre-defined predicate libraries, future work aims to incorporate automated predicate generation and optimize scalability for deeper neural networks. By viewing abstraction as a searchable space, Fanoos allows users to "dial in" the exact level of detail needed for trust and certification in ML systems.

Summary updated

Abstract

Machine learning is becoming increasingly important to control the behavior of safety and financially critical components in sophisticated environments, where the inability to understand learned components in general, and neural nets in particular, poses serious obstacles to their adoption. Explainability and interpretability methods for learned systems have gained considerable academic attention, but the focus of current approaches on only one aspect of explanation, at a fixed level of abstraction, and limited if any formal guarantees, prevents those explanations from being digestible by the relevant stakeholders (e.g., end users, certification authorities, engineers) with their diverse backgrounds and situation-specific needs. We introduce Fanoos, a framework for combining formal verification techniques, heuristic search, and user interaction to explore explanations at the desired level of granularity and fidelity. We demonstrate the ability of Fanoos to produce and adjust the abstractness of explanations in response to user requests on a learned controller for an inverted double pendulum and on a learned CPU usage model.
BibTeX
@inproceedings{bayani2020fanoos,
  author = {David Bayani and Stefan Mitsch},
  title = {{Fanoos: Multi-Resolution, Multi-Strength, Interactive Explanations for Learned Systems}},
  booktitle = {VMCAI},
  year = {2020},
  doi = {10.1007/978-3-030-94583-1\_3},
}
Figure