TL;DR
This paper introduces Mechanistic Anomaly Detection (MAD) as a way to help supervise increasingly capable Large Language Models (LLMs). By analyzing internal model features, MAD aims to flag potentially anomalous training signals that might otherwise go unnoticed, helping to identify and address subtle supervision failures. While MAD shows promise, its effectiveness is highly dependent on the specific model and task, suggesting further advancements in detection and evaluation are needed for high-stakes applications.
What Problem Does This Paper Address?
As LLMs become more powerful, supervising them effectively becomes a significant challenge. These models can learn and act on complex, subtle cues that are not apparent to human supervisors. This can lead to "supervision failures," where models exhibit seemingly desirable behavior that is actually based on flawed reasoning or unintended biases. The authors propose Mechanistic Anomaly Detection (MAD) as a method to augment human oversight by identifying these anomalous behaviors without needing to fully understand the model's internal reasoning process. The goal is to flag instances that deviate from expected patterns, allowing for deeper investigation or filtering.
Methods
The core of the paper is the exploration of Mechanistic Anomaly Detection (MAD) techniques. The authors train "detectors" using internal model features (activations, attribution patching outputs, etc.) to distinguish between normal and anomalous behavior. They experiment with various feature extraction methods and scoring rules. Key techniques include:
- Activation Features: Using internal activations from different layers of the model.
- Attribution Patching: Measuring the impact of ablating model activations on the final output to identify influential internal interventions.
- Probe Shift: Estimating how much intervening in the network affects the accuracy of a probe trained to predict model outputs.
- Misconception Contrast Probing: Training a probe to distinguish between correct and incorrect statements to identify anomalous operational modes.
- Confidence Estimation via Iterative Prompt Rephrasing: Assessing model sensitivity to prompt variations.
- Sparse Autoencoders (SAE) and Normalizing Flows: Using these techniques to extract features from model activations.
They also explore two settings: "Offline Settings" where training data is labeled and test data is unlabeled, and "Online Settings" where the training dataset is limited to labeled trusted data for detector training.
Datasets
The authors utilize modified versions of "quirky" datasets originally introduced by Mallen et al. (2024). These datasets are designed to simulate oversight challenges by training models to exhibit both normal ("Alice") and anomalous ("Bob") behaviors. The datasets cover various topics, including:
- Capital cities of given countries
- Hemispheres of given cities
- Scientific Q&A
- Populations of given cities
- Sentiment detection
- Natural language inference (NLI)
- Arithmetic operations (addition, subtraction, multiplication, modular addition, squaring)
A key modification made by the authors was the introduction of a larger set of "Alice-like" and "Bob-like" names to make the task more challenging and less reliant on simple name labels.
Key Insights and Ideas
- Model-Dependent Detection: The effectiveness of anomaly detectors varies significantly across different language models. This suggests that a one-size-fits-all approach to MAD is unlikely to work.
- Task Specificity: Detection efficacy is highly dependent on the task. Arithmetic tasks, for instance, generally yield better discriminative performance compared to other types of tasks.
- Behavior Strength Matters: Larger deviations in model behavior tend to make anomalies easier to detect, though this is not a universal rule.
- Attribution vs. Activations: While activation-based detectors and attribution-based detectors often show correlated performance, attribution-based methods can sometimes succeed where activation-based methods fail, suggesting they capture different aspects of model behavior.
- Layerwise Performance: The performance of anomaly detectors can vary significantly across different layers of a model, with some methods showing stronger performance at deeper layers.
Key Results
The paper reports mixed results for Mechanistic Anomaly Detection.
- Average Performance: On average, anomaly detectors score significantly above chance, but they misclassify many examples. Offline detectors generally perform better than online detectors.
- Model Performance: Anomaly detectors on the Mistral 7B v0.1 model generally performed better than those on the Llama 3.1 8B model.
- Dataset Heterogeneity: There is significant variation in performance across different datasets. Arithmetic datasets tend to show high AUROC scores, while non-arithmetic datasets show more mixed results. For example, Llama 3.1 8B showed uniformly below 0.8 AUROC on non-arithmetic datasets.
- Label Imbalance: Extreme label imbalance in datasets like SciQ and Population presented challenges, with over 99% of labels being "true" in one partition and "false" in another.
- Quirkiness Correlation: Low anomaly detection performance on Llama finetunes was often associated with low "quirkiness" (a measure of how much prompt labels impact model behavior), though exceptions existed.
Limitations and Future Work
The authors acknowledge several limitations and suggest future directions:
- Artificial Datasets: The "quirky" datasets are intentionally artificial. Real-world tasks will differ systematically, and the results indicate that this will likely lead to systematic changes in detector performance.
- Need for Better Benchmarks: To assess the real-world applicability of anomaly detection, benchmarks that are more closely aligned with intended tasks are necessary.
- Model Size Trend: The finding that anomaly detection was worse on the more capable Llama 3.1 8B model than Mistral 7B v0.1 is an inversion of a trend found in previous work (Bai et al., 2022), where larger models made anomaly detection easier. Further experiments are needed to assess if this is a general trend or specific to these models.
- Advances Needed: For MAD to be effective in high-stakes settings, advances in both detection methods and evaluation strategies are likely required.