TL;DR
This paper introduces a novel framework for analyzing the "self-rationalizing" capabilities of Large Language Models (LLMs) by building a statistical surrogate model based on a specific hypothesis about how the LLM solves a task. While their initial attempt to model GPT-3.5's Natural Language Inference (NLI) didn't achieve high similarity, the framework offers a promising way to probe LLM reasoning and improve interpretability.
What Problem Does This Paper Address?
Large Language Models (LLMs) are increasingly used for complex tasks, often generating natural language explanations (NLEs) alongside their predictions. While these NLEs are valuable for understanding model behavior and building trust, their "faithfulness" – how accurately they reflect the model's actual decision-making process – is a significant concern. It's unclear if the explanations are just post-hoc rationalizations or genuine insights into the model's internal workings. This paper aims to address this by developing a method to quantitatively assess the faithfulness of LLM explanations.
Methods
The core of the proposed framework is a Hypothesis-Driven Statistical Surrogate Model (SSM). This SSM is built using a Bayesian Network (BN), which allows for explicit representation of assumed causal relationships between variables.
- Hypothetical Global Explanation (HGE): The process starts with a human-defined hypothesis about how the LLM solves a specific task. For NLI, the hypothesis was that the LLM compares subphrases from the premise and hypothesis, classifying each pair into contradiction, entailment, or neutral, and then uses deterministic rules to arrive at the final prediction.
- Bayesian Network Construction: The HGE is translated into the structure of a BN. Input variables represent extracted subphrases (e.g., subject, verb, object) from the premise and hypothesis, encoded as word embeddings. Hidden variables in the BN represent the pairwise relationships (contradiction, entailment, neutral) between these subphrases. The output variable is the final NLI label.
- Parameter Learning: The parameters of the BN are learned using the Expectation-Maximization (EM) algorithm, trained on data to match the distribution of the hidden variables given the observed inputs.
- NLE Generation: The SSM generates NLEs by translating the internal states of the BN and the learned rules back into natural language templates.
- Comparison: The SSM's predictions and NLEs are then compared to those generated by a target LLM (GPT-3.5 in this case) using both human and automatic evaluations.
Datasets
- e-SNLI: Used for extracting premises, hypotheses, and human-authored NLEs to train and evaluate the SSM. The paper notes that e-SNLI contains annotation artifacts that might influence results.
- GPT-3.5: Used as the target LLM for comparison. The paper used the
text-davinci-003version.
Key Insights and Ideas
- Decomposing LLM Reasoning: The framework allows for breaking down a complex LLM task (like NLI) into smaller, hypothesized pairwise comparisons, making the reasoning process more explicit and manageable.
- Bayesian Networks as Interpretable Models: BNs provide a natural way to encode hypotheses about dependencies and independencies, offering a visually intuitive representation of the assumed decision-making process.
- Faithfulness as a Measurable Quantity: By comparing a hypothesis-driven SSM to an LLM, the paper proposes a method to quantify the "faithfulness" of the LLM's explanations, moving beyond subjective assessment.
- Small vs. Large SSMs: The study explored different complexities of the SSM (small vs. large), finding that the smaller model with more inductive biases performed better, suggesting that over-parameterization or irrelevant connections can hinder performance.
Key Results
- Low Similarity to GPT-3.5: Both human and automatic evaluations showed a low similarity between the SSMs and GPT-3.5's predictions and NLEs. The accuracy and F1-scores for predicted labels were significantly lower for the SSMs compared to GPT-3.5's performance on the gold standard.
- NLE Correctness: Human raters found the factual correctness of the SSM-generated NLEs to be very low compared to GPT-3.5.
- Small SSM Outperforms Large SSM: The "small" SSM, which had fewer connections and more inductive biases, produced NLEs that were more similar to GPT-3.5's than the "large" SSM.
- Structural Alignment: Despite low overall similarity, human raters indicated that a significant portion of GPT-3.5's explanations followed a structure consistent with the paper's hypothesis (relating subphrases).
Limitations and Future Work
- Hypothesis Simplicity: The initial hypothesis about NLI might be too simplistic to capture the full complexity of GPT-3.5's reasoning. Future work could explore more refined hypotheses.
- Implementation Details: Shortcomings in the SSM's implementation, such as the preprocessing pipeline, template generation, and the neural networks used for parameter learning, might have limited performance.
- Uniqueness of Surrogate Models: For a given hypothesis, multiple SSM structures can be derived, leading to uncertainty in faithfulness estimation. Future work could explore methods to estimate this uncertainty.
- Automated SSM Construction: The current process of manually designing SSMs is task-specific. Developing a generalized procedure for automatically constructing suitable surrogate models would be valuable.
- Reproducibility: Reliance on the GPT-3.5 API poses reproducibility challenges. The paper suggests exploring more open-source LLMs for future research.
- Task and Data Limitations: The e-SNLI dataset's potential annotation artifacts might influence results. Exploring more diverse datasets like MNLI and testing for lexical/syntactic cues is suggested. Language adaptation is also a consideration.
Summary updated