Recurrent Concept Tracking for Nowcasting Neighborhood Change Between Censuses

Census-grade measures of neighborhood welfare, such as household income and population, arrive years late. Street-view photographs of the same corner, taken again and again over a decade, show physical change much sooner. Models that read these photo sequences are either opaque or ignore time. We propose a hybrid convolutional-recurrent network (a CRNN: a convolutional image encoder whose per-photo outputs are fed, in date order, into a recurrent network that keeps a running memory). Its only route to the prediction passes through a small set of named physical concepts per time step, such as "active construction," "new storefront signage," "vacant lot," and "repaved road." The recurrent memory is used to see those concepts more reliably across noisy, irregular photos. The welfare estimate is a transparent function of the concept trajectory. An analyst who corrects one concept in one year should see that correction carry forward to every later nowcast.

In two sentences: temporal context helps a vision model recognize what physically changed on a street, but letting a black box turn that memory directly into an income estimate makes the output impossible to audit. We keep recurrence on the perception side, force the prediction through per-year human-readable concepts, and test whether this matches opaque temporal models on tracking income growth while making corrections persist through time. The audit lens here comes from clinical AI, where passive saliency maps are not considered enough for high-stakes use.

Core Claim

The claim is narrow. On longitudinal street-view data, a concept bottleneck placed after a recurrent layer, fed back into it at the next step, should do three things. It should predict physical concepts more accurately than per-photo prediction. It should track neighborhood income and population growth about as well as an unconstrained temporal model. It should also support corrections by a human reviewer that reduce error at later time steps, not only at the corrected step. The third property is what is new. No work in our file combines recurrent state with an intervenable concept layer, so "intervention persistence" has not been measured for any temporal concept model.

Positioning Against Prior Work

The motivating evidence comes from urban sensing. CityPulse built a 16-year street-view series over 931 US locations. Its Siamese DINOv2 model, which compares photo pairs, detects structural change points with 88.85% accuracy. Those change points correlate with Seattle median household income growth ($R^2 = 0.19$) and population growth ($R^2 = 0.15$), while building permits show almost no correlation. This establishes that physical change beats administrative records for closing the latency gap. However, CityPulse reasons pairwise, keeps no running state, and outputs a binary change signal with no explanation of what changed. We reuse its data and its evaluation against income growth, and differ in carrying memory across the whole sequence and explaining through named concepts.

Two static pipelines argue for explicit intermediate concepts. Generating Interpretable Poverty Maps counts detected objects in satellite images and regresses consumption with gradient boosting, beating end-to-end transfer learning ($r^2$ 0.539 vs 0.41). CityLens finds that scoring visual attributes and then regressing beats direct vision-language prediction. Both are single-snapshot and explain post hoc. Predicting Livelihood Indicators from Community-Generated Street-Level Imagery uses segmentation classes on crowd-sourced photos but discards timestamps. Its segmentation classes supply a vocabulary from which we draw weak concept labels.

The method comes from interpretable medical modeling. Concept Bottleneck Models route predictions through annotated concepts. In their knee X-ray experiments, correcting two concepts cut error from above 0.4 to about 0.3, and the concept model resisted background shift where the black box failed. The same paper warns that unconstrained side channels let models bypass concepts, so our label head sees concepts only. Learning Optimal Summaries of Clinical Time-series with CBMs extends concepts to ICU series but builds them from handcrafted summaries (mean, slope, variance). Its roughly 0.73 AUC matched a black-box LSTM, which warns us that handcrafted temporal summaries may be enough. We treat that as a baseline to beat, not a strawman. Interpretability for Time Series Transformers using a Concept Bottleneck Framework aligns transformer layers with generic concepts (autoregression, time of day) and has no image encoder or intervention.

A constraining result is Benchmarking Deep Learning Interpretability in Time Series Predictions. It shows gradient saliency on recurrent models smears importance across all features at a salient time step. Its Temporal Saliency Rescaling (TSR) raises precision-recall area from 0.331 to 0.399 on its synthetic Middle Box benchmark. The same paper shows that masking-based evaluation is unreliable on real data. We therefore evaluate explanations only where ground truth is known. On architecture, Convolutional Recurrent Neural Networks for Small-Footprint Keyword Spotting shows a compact CRNN (about 229,000 parameters) halving false rejections versus a parameter-matched CNN. That supports the design choice without proving it transfers to images.

Research Program

The model encodes each dated photo of a location with a convolutional network. A recurrent unit (a GRU, a gated recurrent network) reads the sequence of encodings and outputs a probability vector $c_t$ over $K$ concepts at every photo date $t$. The predicted concepts $c_t$ are concatenated to the next step's input, so the memory is conditioned on its own concept readings. The welfare head is deliberately simple:

$$\hat{y}_t = w^\top c_t + v^\top (c_t - c_{t_0}) + b,$$

Here $\hat{y}_t$ is the nowcast (for example, log income change since the last survey at date $t_0$). The weight vectors $w$ and $v$ are indexed by concept and read directly as "value of having" and "value of gaining" each concept, and $b$ is a bias. Because the head is linear in concepts, every nowcast decomposes exactly into per-concept, per-year contributions, with no saliency method needed.

Training is joint, with a concept loss on the photos that have concept labels plus a regression loss only at dates where survey ground truth exists. This mirrors the latency setting: concepts are observable every year, but welfare labels are sparse in time. At test time, an intervention replaces $c_t$ with a human-verified value. Because $c_t$ feeds the next recurrent step, the correction also changes $c_{t+1}, c_{t+2}, \dots$.

Concept labels come in two tiers. Weak labels for roughly a dozen change-relevant concepts come from segmentation classes of the kind used by Lee et al. and from CityPulse's change-point annotations. A human-verified subset of a few hundred sequences serves for evaluation and intervention experiments. Concept completeness is a real risk. We measure it by fitting a black-box residual model on top of the concepts and reporting how much variance it recovers; a large residual means the concept set is missing something.

Baselines

The do-nothing reference is persistence: predict zero change since the last survey, equivalently carrying the last census value forward. Any method that cannot beat persistence on held-out years has not reduced latency at all. Alongside it we report building-permit counts. CityPulse reports these as near-zero in correlation, and reproducing that on our split checks the pipeline.

The strongest existing temporal method is CityPulse's Siamese DINOv2 change detector. We use its detected change points per location as features in the same regression onto income growth. The reported figure is $R^2 = 0.19$ for Seattle median household income growth (and 0.15 for population) in CityPulse. We rerun it on our identical splits rather than quoting it, because our held-out cities and years differ. Beating it means matching or exceeding its $R^2$ on the same splits, within a bootstrap confidence interval over locations. We would not claim to beat it on accuracy alone; matching it while adding decomposable, intervenable outputs is the actual bar.

The strongest simple competitor is a per-photo concept pipeline with handcrafted temporal summaries, in the style of Ayush et al. combined with Wu et al. The same convolutional encoder predicts concepts independently for each photo, the concept series are summarized by last value, mean, slope and max-minus-min, and gradient-boosted trees or ridge regression map summaries to income growth. It gets the same encoder, same concept labels, and the same hyperparameter search budget, measured in GPU hours and number of configurations.

The strongest opaque temporal model is the same CRNN with the bottleneck removed, regressing income directly from the recurrent state, explained post hoc with TSR. This tells us the accuracy cost of the bottleneck.

Two ablations isolate the key ingredients. Removing the recurrence while keeping per-step concepts and the same linear head tests whether memory helps concept perception. Removing concept feedback into the recurrent input tests whether it is what makes interventions persist. The proposal succeeds only if the full model beats the no-recurrence ablation on concept accuracy and the no-feedback ablation on post-intervention error at later years.

Why not just…?

Why not just predict concepts per photo and fit gradient boosting on their summaries? This is the most serious threat. Wu et al. found handcrafted summaries matched an LSTM on ICU data, so our recurrence may add nothing to final income accuracy. Our bet is narrower. Street-view sequences are irregular, with occlusions, seasonal foliage, and differing camera positions, so per-photo concept predictions will flicker. Memory should suppress that flicker and raise concept accuracy, especially for gradual concepts like construction. If the per-photo pipeline matches us on both concept accuracy and income $R^2$, the recurrent contribution is refuted. In that case we would report only the intervention and feedback findings, if those survive.

Why not just use a large vision-language model to describe each photo and regress? CityLens shows zero-shot vision-language models give poor or negative $R^2$ on direct numeric estimation, and CityRiSE reaches only $R^2 = 0.286$ on unseen cities after reinforcement fine-tuning. Both work on static images, and their free-text reasoning is not an intervenable variable. They could still be a strong concept labeler. We will test a vision-language model as an alternative source of weak concept labels, which strengthens rather than replaces our design.

Why not just explain a black-box temporal model with TSR? TSR improves attribution, but Ismail et al. show that on real data there is no ground truth to confirm it. A saliency map also offers no way for a reviewer to correct the prediction. Our semi-synthetic experiment below compares TSR's recovery of the true driver against our concept decomposition directly. The intervention experiment then tests something saliency cannot do at all.

Proposed Experiments

The primary evaluation uses the CityPulse street-view series with income and population growth targets. We train on four cities and test on the fifth, rotating, and additionally hold out the most recent years within training cities. We report $R^2$ on income and population growth against all baselines. We also report error as a function of years since the last survey, which is the quantity latency reduction is supposed to improve. Concept accuracy is measured against the human-verified subset.

The intervention experiment replaces the model's concepts at one chosen year with human-verified values and measures error reduction at that year and at each later year. The expected result is that error drops with feedback and decays back toward the uncorrected error without it. This is the temporal analogue of Koh et al.'s knee X-ray intervention curve.

For explanation fidelity, we build semi-synthetic sequences. Real CityPulse locations get labels generated from a known concept trajectory, for instance income rising two years after construction concepts switch on. We then score which of our linear decomposition, TSR on the black-box CRNN, and plain gradients recovers the true concept and year, using precision-recall area as in Ismail et al.

The decisive checks are three. First, the full model must beat the no-recurrence ablation on concept accuracy for gradual concepts. Second, interventions must reduce error at years after the corrected one, and only when feedback is present. Third, our income $R^2$ must fall within the confidence interval of the black-box CRNN and the rerun CityPulse detector. Failing the first check means memory is unnecessary. Failing the second means the auditability claim collapses into a static bottleneck. Failing the third means the concepts are incomplete, and the residual analysis will tell us which physical changes they miss.

All ground truth here is US data. Because CityLens documents sharp degradation in cities like Nairobi and Mumbai, we make no claim about the Global South. Extending to timestamped crowd-sourced photos matched to household surveys is a follow-up that this design would make possible.