Models trained on old data get worse on new data. When that happens, practitioners usually learn only that accuracy fell, not which part of the model went stale. An inherently interpretable model can help here. A concept bottleneck model makes its prediction in two stages: it first predicts a small set of human-readable intermediate facts ("this visit is a commute," "it is a weekday evening," "the user is near a transit hub"), and then a simple, usually linear, layer maps those facts to the output. This two-stage structure lets us ask a question a black box cannot answer. Did the world change how those facts show up in the raw data, or did it change what the facts imply for the outcome? We propose to measure obsolescence separately in each stage, to test whether this split correctly predicts which cheap repair will restore accuracy, and to track how the accuracy cost of interpretability changes as training data ages.
The opportunity comes from two observations in the file that have not been connected. On the interpretability side, concept bottlenecks degrade less than end-to-end models under some shifts. On CUB+Places, a standard model's error rises to 62.7% while bottleneck models stay at roughly 48–49% (Concept Bottleneck Models). On the obsolescence side, real longitudinal benchmarks document decay without explaining it. For example, Massive-STEPS shows next-place prediction getting worse from 2012–13 to 2017–18 across 15 cities (Massive-STEPS), and the concept-drift survey literature names "where" and "why" as the underdeveloped half of drift monitoring (One or two things we know about concept drift, Part B).
Core Claim
In a concept bottleneck trained on one time period and deployed on a later one, the loss of accuracy can be split into two parts. The first part is caused by the concept predictor going stale. The second part is caused by the concept-to-label head going stale. We claim three things about this split. First, it is stable and can be measured with a small labeled sample from the new period. Second, it correctly predicts which minimal repair recovers accuracy: refitting only the head, refitting only the concept predictor, or adding a new concept. Third, its per-concept breakdown agrees with drift that we inject deliberately and know the answer to.
Write $g_t$ for the concept predictor and $f_t$ for the head fit on period $t$. Write $L_s$ for task loss on period $s$. With old period $o$ and new period $n$, define
$$\Delta_{\text{head}} = L_n(f_o \circ g_o) - L_n(f_n \circ g_o), \qquad \Delta_{\text{enc}} = L_n(f_o \circ g_o) - L_n(f_o \circ g_n),$$
where $f_n$ and $g_n$ are refit on a small labeled sample from period $n$ while the other stage stays frozen. $\Delta_{\text{head}}$ is how much accuracy comes back from updating only the reasoning layer. $\Delta_{\text{enc}}$ is how much comes back from updating only the perception layer. Applying the same quantities to individual concepts (the per-concept prediction error of $g_o$ on period $n$, and the change in each head weight) produces a ledger that says, for example, "the 'evening leisure' concept is still detected correctly, but its weight toward nightlife destinations halved."
This differs from the closest work in specific ways. On the Impact of Temporal Concept Drift on Model Explanations passively measures how token-level rationales hold up across chronological text splits. It finds that faithfulness can even rise under drift, which is a warning that rationale metrics alone can mislead. It does not decompose the drift or link a diagnosis to a repair, and it does not study concept bottlenecks. CONDA adapts a bottleneck at test time on static benchmarks such as Waterbirds and CIFAR-C. It treats aligning concept scores, refitting the head, and adding residual concepts as one bundle, and it never asks which component the shift actually hit. CLG-CBM tracks concept weights as new classes arrive, which is class-incremental learning rather than ageing of existing classes. CODA forecasts raw feature correlations for black-box predictors. Our ledger works at the concept level, where the number of dimensions is small.
Research Program
The main testbed is Massive-STEPS, because its two epochs, five years apart, across 15 cities give natural ageing with large differences between cities. The task is to predict the next place category, not the specific place, because place identities themselves turn over across five years. The concepts can be computed deterministically from metadata the benchmark already supplies: Foursquare category of the current and previous stop, hour band, weekday versus weekend, dwell-time band, distance from the user's home centroid, whether the stop is near an OpenStreetMap transit feature, and a handful of composite activity concepts such as commute, errand, and leisure, defined by transparent rules. Because these concept labels come from rules rather than annotators, they mean the same thing in both epochs. This removes the usual objection that concept supervision is expensive or inconsistent over time. The concept predictor $g$ is a sequence encoder over the raw check-in trajectory. The head $f$ is a sparse linear layer. Both are trained jointly in the style of Concept Bottleneck Models.
The second testbed is the chronologically split text tasks from the temporal-drift explanation study: AGNews, FactCheck, Yelp, and the Amazon categories with their synchronous and two asynchronous splits. Here concepts are generated and assigned by a language model, following StructCBM, whose automatic concept labels agreed with humans 96.43% of the time. This testbed checks whether the ledger works when the concepts are learned rather than computed by rule. It also lets us confront that study's "faithfulness paradox" directly. If the paradox holds, the faithfulness of a black box's rationales should rise or stay flat on the asynchronous splits while our ledger shows a head that has measurably changed.
The program has three parts. The first is a measurement of how the accuracy cost of interpretability ages. For each city or text task, we compute the gap between the bottleneck model and an unconstrained model with the same encoder backbone, first on held-out data from the training period and then on the later period. A shrinking gap would mean the cost of interpretability is partly an artifact of evaluating only on fresh, same-period data.
The second part is the ledger itself. We estimate $\Delta_{\text{head}}$ and $\Delta_{\text{enc}}$, along with their per-concept versions, at labeled-sample budgets of 1%, 5%, and 20% of the new period. We report how much the estimates vary across resamples.
The third part is validation against known answers. In the old period, we inject two kinds of drift by construction. Head drift changes the mapping from concepts to labels: for one city, we reassign which activity concepts lead to which next categories, leaving the trajectories untouched. Encoder drift perturbs the raw inputs so that a concept looks different but implies the same outcome: we shift timestamps by a fixed offset, mimicking a shift in daily routine, while keeping the labels consistent with the concepts. A correct ledger must attribute each injection to the right stage and the right concept.
Baselines
The naive reference point is the frozen old-period model, of both the bottleneck and black-box kinds, applied to the new period without changes. This fixes the size of the decay that every repair has to recover. Massive-STEPS reports the decline only qualitatively. We found no reported result on Lacuna for absolute next-POI accuracy on either epoch, so this number will be produced by us.
The strongest black-box competitor on accuracy is the Spatio-Temporal Hypergraph Convolutional Network (STHGCN), which the Massive-STEPS paper names as its top supervised model (Massive-STEPS); no reported result found on Lacuna for STHGCN's numeric accuracy. We will run STHGCN with three update strategies: frozen, fully fine-tuned on the same new-period labeled sample, and with only its last layer refit on that sample. It gets the same hyperparameter search budget, measured in GPU-hours, as our model. For our proposal to win on accuracy, its best repair has to match the best STHGCN repair within the variance across resamples, at the cost of a sparse linear head. Our main claim does not require beating STHGCN outright.
The strongest existing interpretable method under distribution shift is CONDA, which reports a 28% accuracy gain from adapting a deployed bottleneck on Waterbirds. We will apply its score alignment, head refitting, and residual-concept steps to each new period with its own unlabeled data and pseudo-labels. We will also give it the same small labeled sample our method uses. The comparison has two criteria. On accuracy, a ledger-guided minimal repair should match CONDA's full bundle. On interpretability, the ledger-guided repair should change fewer concepts and add no residual concepts without names.
The key ablation removes the diagnosis. Instead of choosing the repair the ledger indicates, it always applies the same fixed repair: head only, encoder only, or both. If ledger-guided selection does no better than "always refit both" on accuracy per labeled example, the decomposition has no practical value beyond its explanatory content, and we will report that. A second ablation computes the ledger from a black box with a linear probe trained to predict the concepts. This tests whether an actual bottleneck is needed or whether probes suffice.
Why not just…?
Why not just refit the last layer of the black box on fresh data? This is the real threat to any accuracy claim, and we expect it may well match a refit bottleneck head on accuracy for a given number of labeled examples. That is why accuracy recovery is not our central claim. A refit black-box last layer tells you that something changed. It cannot tell you whether a concept is now perceived differently or now means something different, because its features have no names. The probe ablation measures this directly. If concept probes on the black box produce an equally accurate ledger on the injected-drift checks, then a true bottleneck is unnecessary, and the proposal reduces to "use concept probes as drift monitors," which is still a useful result.
Why not just run post-hoc attributions on the black box at each period? The temporal-drift explanation study found that post-hoc attributions behave inconsistently across chronological splits, and that faithfulness can increase even as the model's reasoning shifts. An increase in faithfulness is therefore not evidence that nothing changed. On the injected-drift checks, we will compare attributions on STHGCN, computed with Integrated Gradients and scaled attention (the most temporally robust method in that study), against our ledger. The comparison asks which one points to the injected concept.
Why not just run a statistical drift detector on the raw inputs? Detectors such as Maximum Mean Discrepancy say that the input distribution moved. The drift survey documents that their localization degrades when features are correlated (Part B survey), and they are blind by construction to head drift, which leaves inputs unchanged. Our head-drift injection is designed to separate them: a raw-input detector should raise no alarm there, while $\Delta_{\text{head}}$ should be large.
Why not just apply CONDA to each period? CONDA is the closest method, and on accuracy it may tie us. Its residual concepts, however, absorb exactly the variance that the ledger is meant to attribute, and they carry no names. The comparison above reports both accuracy and how many unnamed residual concepts CONDA had to add, period by period.
Proposed Experiments
The first decisive check uses injected drift on Massive-STEPS. For each injection type, across five cities and three seeds, the ledger must assign the larger share of the decay to the correct stage. It must also rank the injected concept within the top three by per-concept change at the 5% labeled budget. If it fails this check, the claim is wrong.
The second check uses natural drift. On the 2012–13 to 2017–18 split, the repair recommended by the ledger should recover at least as much accuracy as the "always refit both" ablation, at equal or smaller labeled budget, in most cities. We also test whether the ratio of head-driven to encoder-driven decay correlates with the city's category entropy. Massive-STEPS reports that category entropy correlates with accuracy at $r=-0.646$, and the natural question is whether the decay it signals sits in the head or the encoder.
The third check is the ageing curve for the accuracy cost of interpretability, measured on both testbeds. We report the bottleneck-versus-black-box gap on the training period and on each later split. The hypothesis, motivated by the CUB+Places result, is that the gap narrows as data ages. A gap that stays flat or widens is a publishable negative result that limits how far that robustness finding can be generalized.
The final check uses the text splits. We test whether ledger-identified head changes line up with the qualitative vocabulary shifts that study reports, such as fact-checking salience moving from "web" and "site" to "video" and "social." This gives an external sanity check on what the ledger says.
References
- Concept Bottleneck Models
- Massive-STEPS: Massive Semantic Trajectories for Understanding POI Check-ins - Dataset and Benchmarks
- One or two things we know about concept drift - a survey on monitoring in evolving environments. Part B: locating and explaining concept drift
- On the Impact of Temporal Concept Drift on Model Explanations
- CONDA: Adaptive Concept Bottleneck for Foundation Models Under Distribution Shifts
- Language Guided Concept Bottleneck Models for Interpretable Continual Learning
- CODA: Temporal Domain Generalization via Concept Drift Simulator
- Adaptive Concept Discovery for Interpretable Few-Shot Text Classification