This paper, "Auditing Multi-Proxy Explanations of Web User Preferences," tackles a fundamental problem in modern personalization: how do we know if our explanations for user behavior are actually about the user, or just artifacts of how we measure them?
TL;DR
The authors present a robust framework for auditing explanations derived from multiple "proxies" (like behavior, ratings, or LLM tags). They prove that a strict majority of "clean" proxies is necessary and sufficient to recover the true underlying preference signal, and they introduce a system for "abstaining" when the data is too noisy to justify a conclusion.
What Problem Does This Paper Address?
Web systems are constantly trying to figure out what you like. They look at your clicks (behavior), your stars (ratings), and even use Large Language Models (LLMs) to build a "profile" of you. Each of these is a proxy for your actual, latent preference.
The problem is that explaining a proxy often reveals more about the measurement process than the preference itself. For example, a behavioral proxy might say you "prefer Jazz" because of the average price of the albums you buy, while an LLM might say it's because of the tone of your reviews. Is price sensitivity a core part of your "Jazz preference," or just a quirk of how the purchase data was collected?
Without a way to separate the shared signal from proxy-specific bias (contamination), we risk building explanations based on measurement artifacts.
Methods
The authors propose an attribution-level contamination model. They assume that a proxy's explanation (attribution) is a mix of the true latent signal (scaled by some factor) and proxy-specific "noise" or bias.
Key components of their approach include:
- Coordinatewise Median Aggregation: Instead of averaging explanations, they take the median across multiple proxies. This is a classic robust statistics technique, but here it's applied to the high-dimensional space of feature attributions.
- Identification Boundaries: They prove a "strict majority" theorem. If more than 50% of your proxies are "clean" (uncontaminated for a specific feature), the median will correctly recover the direction and support of the true preference.
- Compatible-Target Sets: When there is uncertainty about which proxies are clean or what their relative scales are, the framework doesn't just give a single answer. It provides a set of all possible true attributions that are compatible with the observed data.
- Abstention: If the "compatible set" includes zero, it means the data doesn't strongly support a positive or negative preference. In this case, the system abstains from making a claim—a crucial safety feature for reliable AI.
Datasets
The researchers evaluated their framework using two main sources:
- Controlled Known-Truth Benchmark: A synthetic dataset where the "ground truth" preferences and the exact amount of proxy contamination are known. This allowed them to map the theoretical boundaries of their method.
- Amazon Reviews 2023 (CDs & Vinyl): A real-world study using 5,000 users. They used three proxies: user behavior, and profiles generated by two different LLMs (Qwen2.5 and Llama-3). They evaluated how well these proxies predicted "future" ratings.
Key Insights and Ideas
-
The 50% Sharp Boundary: The most striking insight is the "phase transition" shown in their experiments. As long as the fraction of contaminated proxies is below 0.5, the error remains low. Once you hit that 50% mark, the ability to recover the true signal vanishes.
Figure 1: Controlled strict-majority phase diagram at 𝜆= 1. The vertical axis reports bias leakage: mean absolute attribution on the designated bias coordinate divided by the average mean absolute attribution over the two true driver coordinates. It is shown for the signed local median at every contamination count and 𝐾∈{3, 4, 5, 7} in the additive and logistic-link DGPs. Bands are pointwise 95% Student-𝑡intervals over ten exploratory seeds. Leakage remains small when 2𝑚< 𝐾and rises under the shared-direction construction when the uniform guarantee ends. The theorem is worst-case: losing the guarantee does not require every beyond-boundary instance to fail. -
Abstention as a "Readiness Audit": The authors argue that we shouldn't always force an explanation. Their "compatible-set" audit acts as a "measurement-readiness" check. If the set is too large or contains zero, the system is essentially saying, "I don't have enough clean data to be sure."
- Scale Uncertainty: Even if you know the direction of a preference, you can't know its exact magnitude without "anchoring" the proxies to a known scale. Without this, you can say what someone likes, but not necessarily how much more they like it compared to something else.
Key Results
- Synthetic Success: In the controlled benchmark, the median-based audit achieved 100% coverage of the true signal and made zero "false sign" decisions (claiming a preference was positive when it was actually negative) when given the correct error budgets.
- Amazon Real-World Performance: On the Amazon dataset, the median of the three proxies (behavior + 2 LLMs) consistently reduced sign errors compared to using behavior alone.
- The Reality Check: The audit of the Amazon data revealed that under realistic noise budgets, the system should actually abstain over 98% of the time. This highlights that while multi-proxy systems are better than single-proxy ones, current web data is often too noisy to support highly specific "why" conclusions.
Limitations and Future Work
- External Anchors: The framework relies on external data to calibrate the "scale" of different proxies. If these anchors are weak, the magnitude of the recovered preference will be inaccurate.
- Shared Bias: The "strict majority" rule assumes that proxy biases are somewhat independent. If all your proxies (e.g., different LLMs) share the same underlying training data bias, they might all be "contaminated" in the same way, breaking the majority rule.
- Causality: The authors note that their method identifies "marginal model attribution," which isn't the same as a true causal effect. Future work could integrate these robust aggregation techniques with more formal causal discovery methods.
Summary updated