City activity repeats itself. A business district fills every weekday morning and empties every evening, and a hospital stays busy around the clock. Neighborhoods that do the same job in different cities, such as Trastevere and Soho, show almost the same weekly pattern of activity. Large language models (LLMs) that forecast urban quantities like taxi demand ignore this repetition. They feed each region's history to the model as its own flat run of tokens. Memory and latency then grow with the number of regions times the length of history. The redundancy is obvious to a person, yet the model pays for it in full.
We propose to pay for the repetition once. Before the LLM sees anything, the history of every region is decomposed into a few shared weekly rhythms, each region's mix of those rhythms, and a sparse set of "surprise" entries where observed activity departs from its expected rhythm. The model reads one token per shared rhythm, one per region, and one per surprise, instead of a token stream per region. The bet is that this structured prompt matches flat-prompt forecast accuracy at a fraction of the key-value cache. The key-value cache is the per-token memory an LLM keeps during generation, and it grows linearly with prompt length. We also bet that, at the same memory, the structured prompt beats generic cache-compression methods, because those methods cannot see that region 37 and region 112 are the same rhythm with different scales.
Core Claim
In multi-region spatiotemporal prompts, most of the information sits in a low-dimensional, periodic, cross-region structure. A three-level token hierarchy built from that structure is a better compression than token-level key-value cache compression applied after the fact. Concretely, let $X_r(t)$ be the observed activity of region $r$ at hour $t$. We fit
$$X_r(t) \approx \sum_{k=1}^{K} w_{rk}\, B_k(t \bmod 168) + e_r(t),$$
where $B_k$ is one of $K$ shared nonnegative weekly rhythms over the 168 hours of a week, $w_{rk} \ge 0$ is region $r$'s weight on rhythm $k$, and $e_r(t)$ is the residual. The prompt contains $K$ rhythm tokens, $R$ region tokens built from each region's weights $w_{r\cdot}$, and deviation tokens only for the entries where $|e_r(t)|$ exceeds a threshold $\tau$. Prompt length becomes $K + R + D(\tau)$, where $D(\tau)$ is the number of surprise entries above the threshold. A flat encoding instead costs roughly $R$ times the number of history patches per region.
The closest work splits into three groups, and each plays a different role. UrbanGPT supplies the setting and the backbone. It encodes each region's series with a dilated temporal convolution, maps the result into a Vicuna-7B through special history tokens, and puts a regression head on a prediction token. It reports zero-shot gains but treats regions as independent flat token runs and does not consider cache cost. We keep its encoder and head and change only what the prompt contains.
AnchorKV supplies the method template we adapt. It keeps a few full-precision anchor tokens and stores the remaining tokens as projections onto those anchors plus low-bit residuals for high-impact tokens. Its anchors are chosen by sampling and attention scores inside one sequence. Our anchors are periodic rhythms shared across regions and are chosen from domain structure before any token exists. PyramidKV and Ada-KV split a cache budget across layers and across attention heads respectively. Both treat tokens as flat and independent, so they cannot merge two tokens that carry the same rhythm.
The decomposition itself is not new. Zhuang et al. used nonnegative matrix factorization (NMF, a factorization into nonnegative parts that tends to yield interpretable components) to pull recurring rhythms out of sparse visit matrices. Gionis et al. showed that functionally similar neighborhoods across cities have near-identical temporal signatures. That finding is the evidence that cross-region sharing should exist at all. Neither paper connects the structure to LLM inference.
Research Program
The work has four parts. The first is a cheap diagnostic run before anything is built. For NYC-Taxi, NYC-Bike, NYC-Crime and CHI-Taxi, we measure how much variance $K$ shared weekly rhythms explain, and how $D(\tau)$ grows as $\tau$ falls. If taxi inflow is not well explained by a handful of rhythms, the idea is dead and we stop.
The second part builds the encoder. Rhythms come from NMF on training-city history, refit per city, with $K$ chosen on validation. Each rhythm is rendered as a token through UrbanGPT's temporal convolution applied to $B_k$. Each region token combines a learned projection of the weights $w_{r\cdot}$ with a short summary of the most recent days. Each deviation token carries a region embedding, an hour-of-week embedding and the residual value. The backbone and the regression head follow UrbanGPT. Cross-city transfer works because a new city only needs its own NMF fit. A newly opened region with little history can borrow weights from its nearest functional twin, which addresses the cold-start problem noted in the file.
The third part trains the model on the same instruction-tuning data UrbanGPT uses, with the hierarchical prompt in place of the flat one. The fourth part measures accuracy against peak key-value memory, prefill time and decode time as Pareto curves, sweeping $\tau$ and $K$.
Baselines
The naive reference points come at both ends of the cost curve. At the expensive end is full flat UrbanGPT with an uncompressed cache. On zero-shot NYC-Taxi inflow it reports a mean absolute error (MAE) of 6.16 (UrbanGPT). At the cheap end is flat UrbanGPT with history truncated to the most recent day, which uses the smallest cache.
A non-LLM reference is ASTGCN, which UrbanGPT reports at 9.75 MAE on the same zero-shot task. Any compressed LLM that drifts toward that number has lost the reason to use an LLM. We will rerun both UrbanGPT figures in our own pipeline rather than rely on the reported values. No reported result was found on Lacuna for UrbanGPT's MAE on CHI-Taxi, NYC-Bike or NYC-Crime, so those flat-model baselines will be established by our runs.
The strongest competitors are generic cache compressors applied to the flat UrbanGPT prompt: AnchorKV, PyramidKV, and SnapKV with Ada-KV's head-wise allocation. AnchorKV is the most dangerous of these. It reports keeping 99.3% of the uncompressed RULER score at 20x compression on Llama-3.1-70B, against 86.8% for its strongest baseline (AnchorKV). It also avoids the irreversible-eviction failures that hurt the other methods. Ada-KV reports moving RULER from 44.02 to 53.29 at a 20% budget (Ada-KV). Those numbers come from text retrieval, not forecasting, so the comparison here must be run by us. All methods share the backbone, the training data and the hyperparameter search budget. They are compared at matched peak key-value bytes, measured on the same GPU, with prefill overhead reported separately because AnchorKV pays extra prefill time. We beat these baselines if our MAE is lower at equal memory, or if we reach flat-model MAE within a fixed tolerance at lower memory, across at least three of the four datasets.
The key-ingredient ablations remove the cross-region sharing. The first gives each region its own rhythm tokens: per-region NMF with no shared $B_k$, at the same total token count. The second replaces NMF with learned pooling into the same number of tokens. The third drops the deviation tokens.
Why not just…?
Why not just apply AnchorKV or PyramidKV to UrbanGPT and be done? This is the main threat. If such a method reaches equal accuracy at equal memory, our structured prompt is unnecessary. Our reason to expect a gap is that these methods compress after the prompt has been processed, so prefill still pays for the full flat sequence. They also choose anchors within one sequence by attention, not by recognizing that two regions carry the same scaled rhythm. The matched-memory Pareto comparison settles the question directly. We will report it even if we lose.
Why not just shorten or average the history? Truncation and daily averaging are the cheapest compressions. They destroy exactly what matters for forecasting: weekly position, and the departures from routine (events, holidays) that the deviation tokens are meant to keep. The truncated-history baseline and the no-deviation-token ablation measure how much is lost. If averaging matches us, the surprise tokens are not earning their place.
Why not just learn the compression end to end with a pooling encoder? A learned pooler might discover the shared rhythms itself, and it avoids a separate NMF step. The learned-pooling ablation at the same token budget tests this. We expect the explicit shared rhythms to transfer better zero-shot to Chicago, because a new city can be refit without retraining. If learned pooling ties on CHI-Taxi, we would adopt it and keep only the cross-region token-sharing design.
Decisive Checks
The idea stands or falls on four results. First, the diagnostic must show that a small $K$ explains most taxi and bike variance. We expect it to explain much less for crime, which is sparse and weakly periodic, and we predict little gain there. A large gain on crime would point to a leak rather than a success.
Second, at matched key-value memory on NYC-Taxi zero-shot, the hierarchical prompt must beat AnchorKV-compressed UrbanGPT on MAE. Third, the shared-rhythm model must beat the per-region-rhythm ablation. Without that result, the cross-region temporal synchronicity of Gionis et al. is not what is helping. Fourth, on CHI-Taxi transfer, the method must not fall behind flat UrbanGPT. Throughout, we report wall-clock prefill and decode time as well as token counts, since the file notes that savings on paper often disappear without matching memory layouts.
References
- UrbanGPT: Spatio-Temporal Large Language Models
- AnchorKV: Anchor-Residual KV Cache Compression
- PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
- Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
- Understanding People Lifestyles: Construction of Urban Movement Knowledge Graph from GPS Trajectory
- Where Is the Soho of Rome? Measures and Algorithms for Finding Similar Neighborhoods in Cities