Formal Interpretability with Merlin-Arthur Classifiers
Lists
Only you see this
Abstract
We propose a new type of multi-agent interactive classifier that provides, for the first time, provable interpretability guarantees even for complex agents such as neural networks. In our setting, which is inspired by the Merlin-Arthur protocol from Interactive Proof Systems, two agents cooperate to provide a classification: the prover selects a small set of features as a certificate and presents it to the verifier who decides the class. A second, adversarial prover ensures the truthfulness of the system and allows us to connect the game-theoretic equilibrium between the provers and the verifier to guarantees on the exchanged features. We define completeness and soundness metrics to provide a lower bound on the mutual information between the features and the class. Our experiments demonstrate good agreement between theory and practice using neural network classifiers, and we show how our setup practically prevents manipulation.
BibTeX
@inproceedings{waeldchen2023formal,
author = {Stephan Waeldchen and Kartikey Sharma and Max Zimmer and Berkant Turan and Sebastian Pokutta},
title = {{Formal Interpretability with Merlin-Arthur Classifiers}},
booktitle = {International Conference on Learning Representations},
year = {2023},
url = {https://openreview.net/forum?id=7hvbaJ1AbaM},
}Loading related content...