Scoring Delphi Consensus Against Crisis Mobility

When a city is hit by a shock such as a pandemic lockdown or a disaster, people stop commuting downtown and reorganize their movement around home. Trips shrink to the neighborhood, visits to transit hubs fall, and visits to residential areas rise. Which nearby places people still visit depends on which ones they cannot do without: groceries, clinics, pharmacies. Recent work calls this social infrastructure reliance and shows it helps predict who goes where after a disruption. The Delphi method is the standard tool for turning expert judgment into an agreed definition when hard data are missing. It is an anonymous, multi-round survey in which experts rate items, see the panel's summary statistics, and revise. Delphi has one well-known weakness: agreement is not accuracy, and there is rarely any ground truth to check the consensus against.

This note proposes putting the two to work on each other. Experts will run a pre-registered Delphi on which place categories are essential during an acute city-wide emergency, and how visits to each category will change. They will do this before seeing any emergency-period data. A public mobility dataset that contains a real emergency window will then serve as the referee. The study makes two claims. First, crisis mobility data offer a rare test of whether Delphi's iteration and feedback make expert judgment more accurate or merely more uniform. Second, expert-agreed essentiality weights act as a behavioral prior that helps movement models in the regime where data are thin, namely new, rarely visited places in the first days of a crisis.

Core Claim

The proposal makes two testable claims. The first is that the consensus a Delphi panel reaches about category essentiality can be scored against observed crisis behavior. The second is that whether later rounds score better than the first round is an open empirical question, with a plausible negative answer.

The data come from YJMob100K. It records anonymized trajectories on a grid of 500 m cells, with 85-dimensional point-of-interest (POI) category counts per cell. Its second release covers 25k users over 75 days, including a 15-day emergency. This gives an observed change in visit share for every POI category.

The model to be conditioned comes from Learning Individual Movement Shifts After Urban Disruptions with Social Infrastructure Reliance. It conditions a UNet, an image-to-image network here applied to visit grids, on each person's reliance on social infrastructure. Its reliance features rest on a heuristic grouping of POI categories. That paper motivates this work by showing that the reliance signal matters. It also constrains the work, because its own limitations section says the model still smooths over rare, isolated new visits. Those rare visits are exactly where a prior should help, if a prior helps anywhere.

Research Program

The human panel will include 20–30 experts from urban planning, transit operations, emergency management, and public health. Each expert rates all 85 YJMob100K POI categories over three anonymous rounds on two questions. The first question is a 9-point essentiality rating for the first two weeks of a city-wide emergency. The second is a signed forecast of the percentage change in that category's visit share relative to normal days.

Between rounds, panelists see medians, interquartile ranges, and anonymized rationales. This is the controlled-feedback protocol used in the Delphi study on methodological priorities for behavioral trials. Items are retained by Lawshe's content validity ratio, as used for item screening in VISION-Cog:

$$\mathrm{CVR} = \frac{n_e - N/2}{N/2},$$

where $n_e$ is the number of panelists rating an item essential and $N$ is the panel size.

All rounds are frozen and time-stamped before anyone on the team opens the emergency window of the data. YJMob100K masks the city and the type of emergency. That masking is an advantage here, because it prevents experts from recalling the specific event.

An LLM panel runs the same three rounds in parallel. A Delphi with LLM "experts" is cheap and fast, which matters given that human Delphi rounds take weeks. However, Emergence of Biased Consensus in Multi-Agent LLM Debates finds that low-temperature, high-conformity agent groups lock into biased consensus within one or two rounds. It also finds that mixing model architectures, or sampling temperatures between 1.3 and 1.7, softens the effect. We therefore run a homogeneous panel and a heterogeneous panel. A third variant adds trigger-based prompt rewriting in the style of CONSENSAGENT. That method improved CLUTRR accuracy from 0.43 to 0.62. CLUTRR is a symbolic reasoning benchmark, so whether the gain transfers to a geospatial forecasting task is part of what this study measures.

Two analyses follow. The forecast analysis scores each round's category-level forecasts against observed change in visit share. It uses Spearman rank correlation, sign accuracy, and Brier-style calibration on the essential/non-essential call, with the observed label defined as a category whose share does not fall. It then asks whether round 3 beats round 1 and whether the panel median beats the best individual expert. The modeling analysis feeds each set of weights (human Delphi, each LLM variant, and the alternatives below) into the reliance features of the conditioned UNet, holding the architecture fixed. It evaluates next-day visit prediction during the emergency days. Overall accuracy is near ceiling in this task: the published model gets 0.9761 against 0.9722 for a plain UNet. We therefore make recall and area under the precision-recall curve on newly visited cells the primary metrics. Results are stratified by how rare each cell is in the person's history.

Baselines

The naive reference point has two parts. One is a persistence predictor that assigns each person their normal-period visit distribution. The other is the plain UNet trained on pre-event movement only, which the source paper reports at 0.9722 overall accuracy and 0.9732 unvisited accuracy on YJMob100K.

The strongest existing method is the conditioned UNet with its original heuristic reliance grouping, reported at 0.9761 overall and 0.9761 unvisited accuracy in Learning Individual Movement Shifts After Urban Disruptions with Social Infrastructure Reliance. We first reproduce this number on our splits before any comparison. The source reports no precision-recall figures on newly visited cells (no reported result found on Lacuna for this). We will therefore compute that metric for all methods ourselves.

The most serious competitor is a data-learned weighting. In this variant, the 85 category weights are free parameters fit end-to-end, or estimated from the visit-share change on whatever emergency days are in the training split. Every weighting variant gets the same architecture, splits, random seeds, and hyperparameter search budget.

Three ablations isolate the key ingredient. Uniform weights test whether any reliance signal helps. Delphi weights randomly permuted across categories test whether the specific expert assignment matters. Round-1-only weights test whether iteration adds anything over a single blind survey.

Beating the strongest method means a gain in newly-visited-cell precision-recall area that survives a paired bootstrap over users, without loss of overall accuracy. Beating the learned-weight competitor counts only in the low-data regimes described below.

Why not just…?

Why not just learn the category weights from the data? This is the real threat, and we expect it to win once a few emergency days are available for training. The expert prior can only be justified where data are absent or scarce. So the decisive comparison varies how many emergency days enter training (zero, one, three, or seven) and plots both methods' curves. If learned weights match Delphi weights at zero to one days, the modeling claim fails and only the forecast-accuracy contribution remains. We will report that outcome as a result, not hide it. There is indirect support for priors under shift. Modeling Population Mobility Flows needed gravity structure to bring disrupted inter-city flow error from 9.48 (gravity alone) and 5.33 (single-feature model) down to 2.98. That is suggestive but not proof for individual-level grids.

Why not just ask a single LLM, or take a one-shot expert poll? Both are cheaper. The round-1-only ablation and the single-model LLM condition test them directly. If Delphi's later rounds do not beat round 1 on forecast accuracy, iteration buys agreement and not truth. That is a useful negative finding for the many Delphi studies that cannot check against ground truth. It would also echo the conformity lock-in reported for agent debates.

Why not just hold an expert roundtable? Open discussion is fast but lets status and confident voices set the answer, which is what Delphi's anonymity is meant to prevent. We do not argue this point in the abstract. A small roundtable condition producing one group rating per category, scored with the same forecast metrics, would show whether structure matters on this task. We include it only if panel recruitment allows. Otherwise we flag it as untested.

Decisive Checks

The study succeeds in its first claim if Delphi forecasts correlate with observed category shifts well above a model that just carries forward pre-event visit shares. It is informative whichever way the round-3-versus-round-1 comparison falls.

It succeeds in its second claim only if Delphi-weighted reliance features beat both the heuristic grouping and learned weights on newly visited cells when few emergency days are available for training, and if permuted weights erase the gain.

A secondary check compares heterogeneous and homogeneous LLM panels against the human panel. This asks whether the bias-softening effect found in agent debates carries over to a task with an external referee.

The main risks are that YJMob100K's masked event type makes expert forecasts too generic, and that a few hundred thousand emergency-period visits leave learned weights hard to beat. The first risk is partly mitigated by asking experts to forecast for an unspecified acute emergency. The second is exactly what the low-data curve is designed to reveal.