Do Synthetic Shoppers Notice the Tax? Salience as a Realism Test for LLM Consumer Data

Researchers and companies increasingly run AI agents built on large language models (LLMs) through simulated shops. They do this for two reasons: to generate synthetic consumer data in place of expensive human experiments, and to test the agents that will soon buy things on people's behalf. Both uses rest on an unchecked assumption. Either the agents react to prices the way people do, or they react the way a careful total-cost calculator would. Behavioral public finance gives an unusually sharp human reference point for testing this. When sales tax is revealed only after a shopper enters the checkout funnel, people respond to it about one-third as strongly as they respond to the same change in the posted price (Consumer behavior in online shopping is affected by sales tax). This proposal turns that one-third into a test. We build a two-stage checkout in which costs are either posted upfront or disclosed late. We measure the same ratio of sensitivities for LLM agents and for a matched human sample. We then ask how much human data a synthetic agent panel can actually replace when the goal is to estimate a policy-relevant tax response.

The bet is that matching overall choice shares is a weak standard for synthetic consumer data. Agents can match how often people pick the cheaper item while getting the mechanism completely wrong. A salience ratio separates the possible mechanisms. A value near one means the agent computes the full price like the textbook rational buyer. A value near the human one-third means human-like inattention. A value far outside both means some artifact of how the agent reads web pages. Because the salience ratio has a documented human value and a clean counterfactual, it can serve as a structural "Turing test" for synthetic consumers.

Core Claim

We define the salience ratio as

$$\theta = \frac{\partial \Pr(\text{buy}) / \partial \tau}{\partial \Pr(\text{buy}) / \partial p},$$

where $\tau$ is a cost (tax or fee) disclosed only at checkout, $p$ is the posted base price, and both derivatives are measured in dollars at the same total price. We claim three things. First, current LLM shopping agents will not reproduce human $\theta$, and the direction of the error will depend on the model. Second, calibrating agents to match human choice shares on posted prices alone will not fix $\theta$, which would make $\theta$ a held-out realism check that aggregate fit metrics miss. Third, the value of synthetic agent data for estimating $\theta$ can be expressed as a human-sample exchange rate in the sense of On the Equivalency, Substitutability, and Flexibility of Synthetic Data, and that rate will be poor, possibly zero, unless the agents first pass the salience test.

Prior work motivates the first claim from both sides. A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experiments rewrites live shop pages between the site and the agent. It found that 17 LLMs amplify ratings and nudges into choice shifts of 30–80 percentage points (pp), where humans shift only 5–10 pp. It also found that agents decide lexicographically: they compare the most important attribute first and look at the next only when there is a tie. That suggests agents may either over-weight any explicit dollar figure or ignore a late cost entirely once an earlier attribute has decided the choice. That study only manipulated upfront prices and badges, and it never asked whether agent choices could stand in for human data. We add late-disclosed costs and a substitutability analysis.

Research Program

The environment extends the rewriting-proxy idea from Cherep et al. to the OneStopMarket shop. We add a real checkout stage. A product page shows the base price. Adding the item to the cart leads to a checkout page where a state sales tax, a shipping fee, or both are itemized. The agent or human must then confirm or abandon.

In the inclusive condition, the same total cost is shown on the product page. Crossing base price, late cost, and disclosure timing gives the variation needed to estimate $\theta$ by conditional logit, a standard choice model in which the probability of buying depends on a weighted sum of attributes. The design has two variants. In the single-item variant, the decision is buy or abandon, which mirrors the funnel drop-off measured on eBay. In the two-tab variant, an item with a lower sticker price but a higher late fee is compared against an all-inclusive alternative, which mirrors comparison friction.

A third manipulation is borrowed from Comparison Friction: Experimental Evidence from Medicare Drug Plans. That study sent seniors a personalized letter stating their exact savings from switching plans, which raised switching from 17% to 28%. Our version adds a summary panel stating the total-cost difference between the two items. This tests whether reducing comparison effort moves agents the way it moved people.

We run the same model families studied by Cherep et al. across several prompted user profiles. We also recruit a human sample on the identical interface. The human sample is the reference, because the eBay estimate predates the Wayfair ruling and was measured in a different funnel. We treat the eBay one-third as an external sanity check on our human arm, not as ground truth.

Borrowing the logic of Is this Model for Real?, we compare two quantities. The first is the spread of $\theta$ between independent agent populations that differ only in random seed and persona sample. The second is the gap between the agent $\theta$ and the human $\theta$. A synthetic population "passes" only if its gap to humans is no larger than its gap to itself. That paper showed that fit metrics such as RMSE and AUC could not tell whether a knowledge-tracing model's mechanics were real. Its role here is to justify why we test a mechanism-level quantity rather than predictive fit.

Finally, we measure substitutability. Following Chang et al., we fit the choice model on $n_s$ synthetic trials used as a prior plus a fraction of the human trials, and record the error in the estimated $\theta$ and in the predicted demand change under a move to all-inclusive pricing. The exchange rate is the number of synthetic trials worth one human trial at equal estimation error. Chang et al. report about 30:1 for tracking video. We expect behavioral data to be far worse, and a finding that synthetic data biases the estimate, so that the exchange rate is effectively undefined, is a legitimate outcome.

Baselines

The naive reference point is the rational-buyer assumption, $\theta = 1$. Under it, disclosure timing has no effect and synthetic data are unnecessary, because demand depends only on the total price. We run it by predicting human choices from a logit on total price alone. Our agent-based and hybrid estimators must beat it on held-out prediction of the difference between human behavior in the late and inclusive conditions. A second naive reference is the human-only estimator: the same choice model fitted to the human trials alone, with no synthetic data. This is the substitutability baseline. Synthetic data "helps" only if, at a fixed human sample size, adding agent trials reduces error in $\theta$ relative to this estimator.

The strongest existing method is the choice-experiment protocol of Cherep et al.. Its standard of realism is agent-versus-human choice shift in percentage points, where they report human shifts of 5–10 pp against agent shifts of 30–80 pp. We run that protocol unchanged on our checkout pages, with the same models, prompts, and trial budget, and compute its pp-shift realism score. Our proposal beats it in one specific sense: if some agent configurations match human pp shifts on posted-price attributes but fail the $\theta$ test, then the salience ratio exposes realism failures that the existing metric certifies as acceptable. No reported result on a tax or late-fee manipulation exists for this protocol; none was found on Lacuna for ABxLab with shrouded costs.

For the underlying human phenomenon, the reference number is sensitivity to sales tax at about one-third of sensitivity to base price (Einav et al.). For the friction intervention it is the 11 pp switching gain (Kling et al.).

Two ablations isolate the key ingredient. The inclusive-pricing ablation removes the timing difference while holding the dollar amounts fixed, so any remaining effect of $\tau$ on agents reflects formatting rather than disclosure. The cart-memory ablation gives the agent a persistent note containing the posted price when it reaches checkout. WebShop documented that agents become greedy and do not backtrack, with option-selection scores of 38.9 against 73.9 for humans. The memory ablation separates whether a low agent $\theta$ comes from forgetting across pages or from how the model weighs costs.

Why not just…?

Why not just run the human experiment and skip the agents? For estimating human salience, that is the gold standard, and our design includes it. The question we answer is different: whether the cheap synthetic panels now being proposed as stand-ins can be trusted, and at what exchange rate. Cherep et al. ran 80,000 agent trials against 30 humans. That imbalance is exactly why practitioners will be tempted to rely on agents, and the substitutability curve tells them when not to. Separately, agents are becoming buyers themselves. Their $\theta$ therefore matters for merchants and regulators whether or not it matches humans.

Why not just calibrate the agents with personas or prompts until their choice shares match humans? This is the real threat. If prompting agents to match human choice shares on posted prices also brings their $\theta$ to the human value, the salience ratio adds nothing beyond ordinary calibration. We test this directly. We calibrate personas on the inclusive-pricing trials only, then evaluate $\theta$ on the held-out late-disclosure trials. The lexicographic, switch-like response to preference prompts reported by Cherep et al. predicts that calibration will overshoot, but we treat that as a hypothesis, not evidence. If calibrated agents pass, we report that the test is easy to satisfy and shift the contribution to the substitutability estimate.

Why not just compare synthetic and human purchase data with a distributional distance? Rosenberg-Kima and Pardos showed that good predictive fit can coexist with wrong mechanics. Chang et al. showed that synthetic data matching surface statistics can hurt downstream tasks. We still compute the distance between agent and human choice distributions as a diagnostic. The decisive check is whether it ranks agent configurations in the same order as the $\theta$ gap does. If the two rankings agree, the cheaper metric suffices.

Why not just instruct the agent to compute the total cost? For deployed buyers this may be the right fix, and we test it as one of the prompted profiles. It does not help with the synthetic-data use, because an agent told to compute the total is by construction not a model of human inattention. It also may not work: the harmful-failure rates of 4–9% that Amazon-Bench reports during cart operations suggest that stated instructions and executed checkout behavior can diverge.

Proposed Experiments and Decisive Checks

The main experiment runs every model and persona through the full design. It uses products and prices from the 1.18 million-item WebShop catalog, served through the OneStopMarket checkout. Tax rates span the 0–10% range typical of US states, and flat and percentage shipping fees are included. The primary metrics are $\theta$ with bootstrap intervals, abandonment rate at checkout, and the gain in switching from the friction panel.

The first decisive check is whether any agent configuration's $\theta$ interval overlaps the human interval, and whether the human arm itself lands near the eBay one-third. If humans do not show salience in our funnel, the realism test has no target and the design must be revised before anything else. The second is the held-out calibration test described above. The third is the seed-versus-human dispersion comparison. The fourth is the substitutability curve: the error in the estimated $\theta$ and in predicted demand under all-inclusive pricing, as synthetic trials replace 0% to 80% of human trials, compared against the human-only estimator.

The outcome that would most change practice is agents that pass on choice shares but fail on $\theta$, combined with a substitutability curve that rises rather than falls. Together these would show that current synthetic consumer panels systematically distort the answer to a basic tax-policy question.