Introduction
As machine learning (ML) models are increasingly deployed in safety-critical and financially sensitive domains, their "black box" nature becomes a significant liability. While the field of Explainable AI (XAI) has grown rapidly, most existing tools suffer from two primary flaws: they provide static, "one-size-fits-all" abstractions and often lack formal guarantees. To address these challenges, we introduce Fanoos, an interactive framework that allows users to explore model behavior at varying levels of detail and formal "strength."
Methodology
Fanoos treats explanation generation as a search and refinement process. Its core pipeline utilizes a Counter-Example Guided Abstraction Refinement (CEGAR) approach to represent the model's input/output space using hyper-cubes (boxes). These boxes are propagated through the model to identify regions satisfying specific user queries.
Query Types and Strengths
Fanoos supports a variety of question types, allowing users to probe the model for formal guarantees or statistical tendencies. For "hard" questions requiring high confidence, the system employs SAT solvers (like Z3) to provide provably sound descriptions. For "usual" behavior, it utilizes statistical sampling to avoid rare edge cases that might clutter a formal explanation.
Interactive Multi-Resolution Refinement
A key feature of Fanoos is its ability to adjust the resolution of an explanation based on user feedback. Users can request that an explanation be "more abstract" or "less abstract." In response, Fanoos adjusts internal hyperparameters—such as box size or predicate specificity—and re-runs its search. This results in a multi-resolution partitioning of the input space, as shown below.
Experimental Evaluation
The authors evaluated Fanoos on two distinct systems:
- Inverted Double Pendulum (IDP): A complex 7-dimensional robotics control policy trained via reinforcement learning (PPO2).
- CPU Usage (CPU): A 5-dimensional polynomial regression task sourced from the OpenML dataset.
Key Results and Insights
The experiments demonstrated that Fanoos successfully navigates the "abstraction ladder":
- Structural Trends: Requests for "more abstract" explanations led to shorter descriptions with fewer disjuncts and more general terms. Conversely, "less abstract" requests provided high-granularity detail; in the CPU model, this increased the box count by over 8,000%.
- Groundedness and Fidelity: Because explanations are built from predicates tied directly to the model's variables, they are grounded in actual logic rather than post-hoc rationalizations. The use of formal verification ensures that these descriptions are mathematically sound over the defined regions.
- The "Washing Out" Problem: Fanoos employs specific scoring metrics during its predicate selection phase to ensure that granular details are not "washed out" by broader, weaker predicates when a user requests high detail.
Conclusion and Future Work
Fanoos provides a flexible, interactive bridge between high-level human intuition and low-level formal guarantees. While currently reliant on pre-defined predicate libraries, future work aims to incorporate automated predicate generation and optimize scalability for deeper neural networks. By viewing abstraction as a searchable space, Fanoos allows users to "dial in" the exact level of detail needed for trust and certification in ML systems.
Summary updated