Pricing Preference Data That Goes Stale: Exact Valuation and Deletion Through a Closed-Form Preference Layer

Preference data ages. A comparison a user made last spring may reflect tastes they no longer hold, and a data vendor selling such comparisons is selling something whose worth changes month to month. Today these two facts are handled by separate tools. Alignment methods discount old data with a fixed decay rate. Data-valuation methods compute a static price once. Unlearning methods delete data after the fact through slow, fragile gradient updates. This note proposes one object that does all three jobs at once. It is a preference layer, meaning the part of a reward model that turns text features into a preference score, whose weights are the solution of a regularized least-squares problem.

Because that solution depends on the data only through small summary matrices, the layer has three useful properties. We can compute exactly how good the model would be without any vendor's or time window's data, which gives a counterfactual value. We can remove a stale window in one matrix solve, which is exact deletion rather than approximate unlearning. And vendors can hand over noised summaries instead of raw comparisons, so that both the model and the prices it produces carry a formal differential-privacy guarantee. The central empirical bet is that weighting data chunks by their current measured value tracks preference drift better than a fixed time decay. The case where this should show most clearly is when old data stays partly valid and new data is noisy.

Core Claim

Closed-form weight editing treats a linear layer as an associative memory that can be rewritten with linear algebra instead of gradient descent. It already works for erasing concepts. Deep Unlearning projects weights onto the orthogonal complement of a forget subspace found by SVD. Orthogonal Concept Erasure for Diffusion Models and Forget Many, Forget Right erase many concepts at once through rotations or a Sylvester-equation solve. Selective Synaptic Dampening dampens parameters that the Fisher information marks as forget-specific.

These papers also expose a hard limit. Deep Unlearning reports that individual samples within a class cannot be separated, because their activation subspaces nearly coincide. Data valuation, on the other hand, seems to need exactly that per-sample resolution. We resolve the tension by changing the unit of accounting. The unit is a chunk: one provider's comparisons from one time window. Chunks are what marketplaces actually price, since Distributional Shapley and DU-Shapley both value datasets rather than points. Chunks are also what drift acts on.

The mechanism is as follows. Write $\phi(x, y_w) - \phi(x, y_l)$ for the difference between frozen LLM features of a preferred response $y_w$ and a rejected response $y_l$ to prompt $x$. Each chunk $c$ is summarized by two quantities: $G_c$, the sum of outer products of these feature differences, and $b_c$, the sum of the differences weighted by their label margins. The label margins come from one Newton step of Bradley-Terry logistic loss (the standard model in which the probability of preferring one response is a sigmoid of the score gap) around a shared anchor. For a coalition $S$ of chunks with weights $w_c$, the preference head is

$$ \theta(S, w) = \Big(\lambda I + \sum_{c \in S} w_c G_c\Big)^{-1} \sum_{c \in S} w_c b_c , $$

where $\lambda$ is a ridge penalty.

Every counterfactual model, meaning "trained on this coalition only," costs one $d \times d$ solve, where $d$ is the feature dimension. Removing a chunk is an exact low-rank downdate. We call this exactness "closed-form" in the literal sense. For this layer, deleting a chunk, retraining on the rest, and editing the weights are the same operation.

Value is measured against the present. Utility $U_t(S)$ is Bradley-Terry accuracy of $\theta(S)$ on a small held-out window of the most recent comparisons. Each chunk's Shapley value $v_c^t$ under $U_t$ then serves two purposes: it is the chunk's payment, and it sets the chunk's training weight. Chunks with negative current value are deleted outright. The claim is that value-derived weights remove the two weaknesses of fixed decay that Right Now, Wrong Then (NS-DPO) acknowledges. Its best decay rate depends on an unknown total amount of drift. It also cannot tell high-quality old data from low-quality new data.

Privacy follows from the same structure. Each provider adds Gaussian noise to its $(G_c, b_c)$ before release. Everything downstream is post-processing of those noised summaries, so it inherits the provider's differential-privacy budget $\epsilon$ at no additional privacy cost. That covers the weights, the Shapley values, and the deletions. This speaks directly to the open question that ALinFiK leaves: how to price private data without leaking influence information. It also answers the finding in Understanding Data Importance in Machine Learning Attacks that high-value data is especially exposed to membership inference, which is an attacker's test of whether a given record was in the training set.

Relation to the Closest Work

Losing is for Cherishing is the nearest neighbor. It replaces retraining with gradient-ascent unlearning inside Shapley estimation and reports speedups of up to 195 times on GPT-2 XL. It motivates this proposal, but its counterfactuals are approximate, it is weaker at detecting noisy data than true retraining, and it is static in time. Our counterfactuals are exact for the preference layer, and the utility function is re-anchored as preferences move.

Unlearning-based sliding window (UIL) uses influence-function unlearning to drop the oldest chunk of an image stream. It supports the idea that deleting stale data speeds recovery from drift: it recovered in 6.33 chunks against 17 for retraining. It also warns that semantic drift is harder than drift in low-level noise. UIL deletes by age; we delete by measured value, and we do it for human preferences rather than image labels.

Unlearning to Align (U2A) and the negative preference optimization line (NPO, SimNPO) treat unlearning as preference optimization over a fixed dataset. They constrain the design in one respect: U2A's observation that unlearning alone teaches the model what not to say, but not what to say, is why our update both downweights stale chunks and re-solves with fresh ones rather than only subtracting.

Dealer and Selling Privacy at Auction supply the economic framing that combines privacy loss with Shapley-based compensation. They price static data; we price data whose worth decays.

Research Program

The work has three stages.

The first stage builds the preference head and checks that it is adequate on its own terms. We freeze a Llama-class backbone, extract response features, and fit the one-Newton-step ridge head. We then measure how far it falls behind full DPO and behind a fine-tuned reward model on stationary UltraFeedback. If the gap is large, the rest of the program is moot, so this comes first. We also test a two-layer variant, in which the final projection of the reward model is edited in the associative-memory style, to see whether some of the gap can be recovered while the math stays closed-form.

The second stage is temporal valuation. The stream is split into provider-by-window chunks. At each step we compute Shapley values of the chunks by permutation sampling, with each coalition evaluated by the solve above. Weights are set to the positive part of each value, rescaled, and negative chunks are deleted. We will prove a simple fact and test it empirically: under an abrupt preference flip, chunks from before the flip receive negative value on post-flip validation data, so the method reproduces the extreme of NS-DPO's decay without a tuned rate. Under partial reversion or recurring preferences, old chunks regain positive value. No monotone decay can express that.

The third stage is privacy. We add calibrated noise to the chunk summaries and trace how accuracy and value fidelity degrade as $\epsilon$ shrinks. We then attack the released values and edited weights with membership inference, following the evaluation style of MUSE. MUSE found that approximate unlearning leaves traces detectable by such attacks, through over- or under-unlearning. Exact downdates should leave none beyond what the noise permits.

Baselines

The naive reference point is the same closed-form head fit on all chunks with equal weight and no deletion. Under a preference flip it should behave like the DPO failure that NS-DPO documents, where reward accuracy falls to about 50–55%.

The strongest existing method is NS-DPO from Right Now, Wrong Then, trained on the same drifting UltraFeedback stream with the same backbone. It gets a search over the discount rate using the same recent validation window our method uses, and a matched GPU-hour budget. The reported figures to beat are post-flip reward accuracy of 60–70%, against about 50–55% for DPO, and an AlpacaEval 2.0 length-controlled win rate of 8.93 against DPO's 7.29. We count it as beaten if our head matches or exceeds its post-flip reward accuracy while using no tuned decay rate. For the win-rate comparison we will use best-of-n sampling from a shared policy, reranked by each reward model. We count it as a clear win if we beat it on the recurring-preference and noisy-recent-data streams described below.

The key ablation isolates value-based weighting from the closed-form machinery. It is the same head with exponential time-decay weights $\gamma^{T-t-1}$ in place of Shapley weights, with $\gamma$ tuned on validation. Any gain over this ablation is attributable to valuation, not to the linear head.

A second ablation replaces Shapley values with exact leave-one-chunk-out values, which are nearly free in this setting.

For personalization we include P-ShareLoRA on Reddit TL;DR with annotator-level providers, under its reported setup. It reaches 74.65% accuracy with Llama-3 8B.

For valuation fidelity we compare against Unlearning Shapley and ALinFiK under matched compute. The ground truth is values from full reward-model retraining on a sampled set of coalitions.

Why not just…?

Why not just use NS-DPO with a well-tuned decay? This is the real threat. If drift in practice is a single change point, tuned decay may be all anyone needs. The exponential-weight ablation on our own head settles it. We include streams where decay is known to be wrong: preferences that flip and then revert, as in the helpful-versus-harmless trade-off of TV-HH, and streams where a recent provider injects low-quality labels. If value weighting fails to beat tuned decay there, we will report that the valuation adds pricing and privacy but no drift benefit.

Why not just learn chunk weights directly by minimizing loss on the recent window? Because each coalition is a cheap solve, this is also easy, and it may track drift as well as Shapley weighting does. We run it as a baseline. It does not yield payments with Shapley's fairness properties, which the marketplace use requires. If it wins on accuracy, the right design is to weight chunks by the learned weights and pay them by Shapley value, and we would adopt that.

Why not just retrain the head from scratch on a sliding window? For this linear head, retraining is exact deletion, and we say so openly. The speed argument from UIL does not apply here. The contribution is instead the choice of what to delete, by measured value rather than by age, and the fact that the retraining runs on noised summaries, so it composes with differential privacy. The sliding-window variant is included as a baseline with its window size tuned.

Why not just prompt the agent with recent history? AcCoRD shows that prompting-based agents satisfy all dynamic preferences less than 30% of the time, and score as low as 0.09 on preferences that are triggered mid-conversation. Prompting also produces no provider-level values. We therefore treat it as complementary rather than competing.

Proposed Experiments

The first decisive check is expressivity. On stationary UltraFeedback, the closed-form head's reward accuracy must land within a few points of a fully fine-tuned reward model. If it does not, we stop and report the negative result.

The second check is drift. We build three drifting streams from UltraFeedback and TV-HH: a single flip, following NS-DPO's change-point protocol; a flip followed by reversion; and a stable core of preferences mixed with noisy recent data. On each we report post-drift reward accuracy against NS-DPO, the decay ablation, direct weight learning, and sliding-window retraining.

The third check is valuation fidelity. We measure rank correlation between our chunk values and retraining-based values, and compute time per value. As a sanity test, we plant a mislabeled or off-topic provider, in the spirit of the injected "Howdy!" data in Howdy!Alpaca, and check that it is assigned negative value and deleted.

The fourth check is privacy. We trace reward accuracy and value correlation as $\epsilon$ ranges from 1 to 8. We also report membership-inference success against the released values and weights, comparing exact downdates with Unlearning Shapley's gradient-ascent deletions. No reported membership-inference number on preference data was found on Lacuna for any method, so this comparison will be a new measurement rather than a match to a published figure.