A company that sells access to a language model pays almost nothing to serve one more copy, so the only way to charge budget users and power users different prices is to sell weaker versions alongside the full one. For simple models this is a solved problem. The seller adds random noise to the weights to make cheaper tiers, and sets prices so that buying many noisy copies and averaging them never beats buying the clean model. For language models, the natural way to make a cheaper tier is different. The seller switches off a capability, such as math or legal reasoning, by suppressing the internal features that carry it. A buyer can then defeat the price menu in a new way. They buy two cheap tiers that each lack a different capability, and send each query to whichever tier still has the skill it needs.
This note proposes to measure that loophole directly and price around it. For any set of cheap tiers, we estimate how much capability a buyer can recover by routing, ensembling, prompting, or lightly distilling across them. We then require that no such combination costs less than the tier it reproduces. Sparse autoencoders split a model's internal activations into many narrow, mostly single-meaning features. They serve two roles here. They are one of the tools for switching capabilities off, and they are a candidate cheap predictor of which combinations will leak. Whether they deserve either role is tested, not assumed.
Core Claim
Model-based Pricing for Machine Learning in a Data Marketplace and its system demo Nimbus sell noise-degraded versions of convex models. They prove that a price curve resists averaging if and only if it is increasing and subadditive in the inverse noise variance. That result supplies our template. It does not transfer to language models, for two reasons. The proof relies on convex losses, and isotropic weight noise in a transformer degrades everything at once rather than any one capability. Dealer extends arbitrage-freeness to a privacy budget, but it still works along a single scalar dial.
Our claim is that, for capability-gated tiers, the right condition is a no-recombination constraint over a menu of tiers $T_1,\dots,T_n$. Let $u(T)$ be the vector of a tier's scores across capabilities, and let $R_{\mathcal{A}}(C)$ be the best score vector that a buyer using attack class $\mathcal{A}$, under a fixed compute budget, can extract from a collection of tiers $C$. A price function $P$ is arbitrage-free against $\mathcal{A}$ when
$$P(T) \le \sum_{T_j \in C} P(T_j) \quad \text{for every tier } T \text{ and every collection } C \text{ with } R_{\mathcal{A}}(C) \ge u(T) \text{ componentwise}.$$
This is the "post-processing arbitrage" notion from On Arbitrage-free Pricing for General Data Queries, restated with models in place of queries. When tiers are Gaussian-noised convex models and $\mathcal{A}$ is averaging, $R$ is known in closed form: $N$ copies cut variance by $N$. The constraint then reduces to Chen et al.'s subadditivity.
For language models, $R$ must be measured. The finding in Steering Language Model Refusal with Sparse Autoencoders shows why it cannot simply be read off tier labels. Clamping a single feature there cut MMLU accuracy by up to 27.30%. Gates have side effects, so combinations can recover more, or less, than the labels suggest. The new contribution is a measured recombination function for real gated language models, together with a pricing step built on it.
Research Program
The program has three parts.
The first part builds tier menus for Gemma-2-2B. Each tier suppresses a subset of four or five capability groups: grade-school math (GSM8K) and clusters of MMLU subjects such as STEM, law, medicine, and humanities. We build each menu with three interchangeable gates. One ablates feature sets chosen from the open GemmaScope autoencoders, using the causal attribution method of Sparse Feature Circuits. One ablates supervised difference-in-means directions. One uses a server-side system prompt. Holding the menu fixed and varying only the gate isolates how much leakage each suppression mechanism causes.
The second part estimates $R_{\mathcal{A}}$ for every collection of up to three tiers under four attack classes:
- per-query routing using a small classifier the buyer trains on their own task labels;
- output voting across tiers;
- adversarial prompting aimed at re-eliciting the suppressed skill;
- LoRA distillation on outputs pooled from the purchased tiers, at fixed small budgets.
The third part prices the menu. Because the menu is finite, revenue maximization subject to the constraint above is a linear program over prices. Simulated buyers then purchase the cheapest collection that meets their needs. Following the instrumental view of An Instrumental Value for Data Production, each buyer's value is the gain on their own task mix rather than an intrinsic score. We draw buyers' task mixtures and budgets from parametric families, as Chen et al. did with synthetic demand.
Selling Data as a Digital Good with Scaling Valuations shows that, when a seller sells a single scalar quantity of a non-rivalrous good, the optimal policy gives every paying buyer the same maximum allocation. Versioning only becomes useful once quality has several dimensions. That is our setting, and it sets a bar: a tiered menu must beat a single full-access price, or the exercise is pointless.
Larger menus have too many combinations to attack exhaustively. For these we test cheap predictors of $R$. One predicts from feature coverage: does the union of features left active in the tiers of $C$ cover the features that $T$'s capabilities depend on? The other predicts from benchmark scores alone, as the per-capability maximum over the tiers in $C$, which is what perfect routing would achieve.
Baselines
The naive reference point is a single flat price for full access. Following the scaling-valuation result, it is the optimal scalar-quantity policy, and every tiered menu is scored as a revenue ratio against it.
The strongest existing pricing method is label-based additive pricing. Each capability gets a base price, and a tier's price is the sum over the capabilities it retains. This is the item-based logic of Deep et al., which is arbitrage-free by construction if the gates are perfect. It is the strongest competitor because it is simple and correct in exactly the case where interpretability works as advertised. We run it on the identical menus and buyer populations, with its base prices tuned by the same LP solver. Our method beats it if, once buyers attack the menu, its realized revenue falls measurably below that of measured-recombination pricing. The gap should be large enough to survive resampling of buyer populations.
The second strong baseline ports Chen et al.'s noise-based pricing to Gemma by adding weight noise at several levels and pricing subadditively in inverse variance. Chen et al. report up to 81.2x higher revenue and 121.1x higher affordability than flat-rate pricing, but on convex UCI models. No reported result was found on Lacuna for noise-based pricing of a transformer. We report its revenue and collateral damage on the same buyers.
For gates, the strongest supervised competitor is difference-in-means. AxBench reports that supervised probes and difference-in-means reach a concept-detection AUROC of about 0.94, against about 0.70 for SAE features. It also reports a prompting steering score of about 0.89. Every gate gets the same labeled data budget for choosing directions and the same evaluation sets.
The key ablation swaps only the leakage predictor: feature coverage versus benchmark-score maximum. Everything else stays fixed.
Why not just…?
Why not just gate capabilities with a system prompt? AxBench shows prompting is the best steering method it tested, so prompt gating may well suppress capabilities most cleanly when no one is attacking it. The weakness is resistance to attack. O'Brien et al. report that multi-turn Crescendo attacks defeat prompt-level defenses, and a paying buyer has every incentive to try. We include prompting as one of the three gates and measure its leakage under the adversarial-prompting attack. If its recovered capability stays low, prompt gating wins and we will say so.
Why not just use supervised directions instead of autoencoder features? This is a real threat on gate quality, given AxBench's detection gap. The proposal does not depend on SAEs being the better gate. The leakage audit and pricing LP work with any gate. What the comparison adds is whether the narrower SAE features cause less collateral damage, and therefore make cleaner menus, despite weaker detection. If difference-in-means gates produce menus with equal or higher arbitrage-free revenue, the recommendation becomes "audit and price supervised gates."
Why not just price by tier labels and assume routing? Label pricing breaks in two ways. It fails if gates leak, and it fails if side effects make the gates superadditive. That second risk is grounded in the 27.30% MMLU drop from a single-feature clamp. The benchmark-score-maximum predictor is exactly the "assume routing" view. If it matches measured $R$ as well as feature coverage does, then autoencoders contribute nothing to prediction, and we will report that.
Proposed Experiments
The first decisive check is whether recombination leaks at all. For each gate, we measure the fraction of a suppressed capability recovered by the best attack within budget, for pairs and triples of tiers. If leakage is near zero for every gate, label pricing suffices and the main claim fails.
The second check compares realized revenue, after buyers attack, across three pricing schemes: flat-rate, label-additive, and measured-recombination. We also report the number of distinct buyers each scheme serves, and collateral damage on held-out MMLU subjects.
The third check measures how well feature coverage and benchmark-score maximum each predict measured recombination. We report rank correlation and the revenue lost when the LP uses predicted rather than measured $R$. We also check whether the predictors hold up when a distillation budget not seen during calibration is used.
Finally, as a sanity anchor, we reproduce Chen et al.'s averaging attack on YearMSD and CovType with logistic and linear regression. The goal is to confirm that our general constraint recovers their subadditive curve where the closed form is known.
References
- Towards Model-based Pricing for Machine Learning in a Data Marketplace
- Demonstration of Nimbus: Model-based Pricing for Machine Learning in a Data Marketplace
- Dealer: An End-to-End Model Marketplace with Differential Privacy
- On Arbitrage-free Pricing for General Data Queries
- Steering Language Model Refusal with Sparse Autoencoders
- AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders
- Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models
- An Instrumental Value for Data Production and its Application to Data Pricing
- Selling Data as a Digital Good with Scaling Valuations
- Revenue Maximization for Query Pricing