Evaluating Concept-based Explanations of Language Models: A Study on Faithfulness and Readability
Lists
Only you see this
Abstract
Despite the surprisingly high intelligence exhibited by Large Language Models (LLMs), we are somehow intimidated to fully deploy them into real-life applications considering their black-box nature. Concept-based explanations arise as a promising avenue for explaining what the LLMs have learned, making them more transparent to humans. However, current evaluations for concepts tend to be heuristic and non-deterministic, e.g. case study or human evaluation, hindering the development of the field. To bridge the gap, we approach concept-based explanation evaluation via faithfulness and readability. We first introduce a formal definition of concept generalizable to diverse concept-based explanations. Based on this, we quantify faithfulness via the difference in the output upon perturbation. We then provide an automatic measure for readability, by measuring the coherence of patterns that maximally activate a concept. This measure serves as a cost-effective and reliable substitute for human evaluation. Finally, based on measurement theory, we describe a meta-evaluation method for evaluating the above measures via reliability and validity, which can be generalized to other tasks as well. Extensive experimental analysis has been conducted to validate and inform the selection of concept evaluation measures.
BibTeX
@inproceedings{li2024evaluating,
author = {Meng Li and Haoran Jin and Ruixuan Huang and Zhihao Xu and Defu Lian and Zijia Lin and Di Zhang and Xiting Wang},
title = {{Evaluating Concept-based Explanations of Language Models: A Study on Faithfulness and Readability}},
booktitle = {Conference on Empirical Methods in Natural Language Processing},
year = {2024},
doi = {10.18653/v1/2024.emnlp-main.36},
}