Share What Intent Does, Not How It Looks: Federated Pedestrian Intent Learning from Hindsight

Cars that drive among pedestrians need to guess, a second or more ahead, whether a person at the curb is about to cross. The best such "intent estimators" learn from large collections of dashcam video annotated by humans. Pooling that video is legally and practically hard, because it shows faces, homes and routines. Federated learning, where each fleet or city trains on its own data and only shares model updates, is the natural fix. It runs into a real problem, though. The visible signs of intent differ from place to place: a glance at traffic means "about to cross" where jaywalking is normal and "waiting" where it is not. What intent leads to, once it exists, is much more uniform. A person who commits to crossing steps off the curb and walks across, unless the car blocks them.

We propose to split the model along exactly that line. Each client keeps a private recognizer that maps its local visual and contextual cues to a latent intent. All clients jointly train a shared outcome model that says what each latent intent implies for the pedestrian's next few seconds, given what the ego vehicle does. Supervision needs no human intent labels. The vehicle sees, in hindsight, whether and how the pedestrian moved, so every client can train locally on its own logs. The shared outcome model anchors the latent space so that "intent code 3" means the same thing in Toronto and in any other city, while the part that sees raw appearance never leaves the vehicle.

Core Claim

For pedestrian crossing intent under city-level heterogeneity, the right half of the model to personalize is the cue-to-intent recognizer, not the predictor head. The right shared component is an action-conditioned outcome model trained from hindsight. Our central prediction is that this split beats FedAvg, FedAvg with local fine-tuning of the output head, and per-client base-rate correction. We will measure this on held-out human intent annotations, on worst-client performance and on calibration. We also predict the advantage reverses in a controlled setting where cues are shared and motion differs. That reversal is the diagnostic that separates our mechanism from generic personalization.

This differs from the closest existing work in specific ways. Personalized Federated Learning of Driver Prediction Models for Autonomous Driving (APFL) trains a conditional variational autoencoder trajectory predictor across a fleet. It decides which weights to personalize from cross-client weight variance, with no structure on the latent space, and it covers only driver styles in simulation. Privacy-preserving and Uncertainty-aware Federated Trajectory Prediction for Connected Autonomous Vehicles runs plain FedAvg on a coordinate forecaster (HiVT on Argoverse) with no latent intent at all. Federated Variational Preference Alignment with Gumbel-Softmax Prior for Personalized User Preferences personalizes a mixture prior over latent preferences to prevent collapse, but only for language-model preference data. None of the three asks which semantic half of an intent model should be shared. None uses hindsight outcomes as the label-free local signal.

Research Program

Each client $k$ trains on its own observation windows. For one pedestrian, an observation window contains appearance crops, bounding boxes, scene context $c$ (crosswalks, signals) and ego-vehicle telematics. The model has three parts. The first is a local recognizer $q_{\phi_k}(z \mid x)$, which maps the observation window $x$ to a distribution over a small discrete set of intent codes $z$. The second is a local prior $p_{\psi_k}(z \mid c)$, which captures how often each intent occurs in a given context in that city. The third is a shared outcome model $p_\theta(y \mid z, e)$, which predicts the pedestrian's future action label and bounding-box track $y$ over the next one to three seconds, conditioned on the intent code and the ego vehicle's future speed profile $e$. The local objective is

$$ \mathcal{L}_k = \mathbb{E}_{q_{\phi_k}(z\mid x)}\big[-\log p_\theta(y \mid z, e)\big] + \beta\, \mathrm{KL}\big(q_{\phi_k}(z\mid x)\,\|\,p_{\psi_k}(z\mid c)\big), $$

where $\beta$ weights how strongly the recognizer is pulled toward the local prior. Only $\theta$ is averaged on the server. Conditioning the outcome model on $e$ matters. Without it, hindsight training would learn "will cross," which is not the same as "intends to cross." PIE explicitly separates the two (Pie: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction). With $e$ as an input, a pedestrian who wants to cross but is blocked by an approaching car can be explained as "crossing intent plus non-yielding ego." The latent therefore has room to represent intent rather than outcome.

The recognizer reuses established designs rather than inventing new ones. We will use the pedestrian-centric graph-plus-recurrent encoder of Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction and the attentive relation network over traffic objects from Coupling Intent and Action for Pedestrian Crossing Behavior Prediction. The second paper also motivates using future actions as a signal for current intent. We move that idea from a centralized auxiliary target to the sole supervision available on each client. A known failure of discrete latent models trained on scarce local data is collapse onto one code (Federated Variational Preference Alignment with Gumbel-Softmax Prior for Personalized User Preferences). We will use a Gumbel-Softmax relaxation and monitor per-client code usage. We expect the shared outcome model to resist collapse by itself: each code must explain distinct futures across all clients, which gives every client a fixed, informative target for its recognizer.

There is also a privacy benefit, which we state as a hypothesis and probe only lightly. The parameters that touch faces and clothing stay on the client. Only the outcome model, which sees codes, boxes and speeds, is transmitted. Layer-wise Characterization of Latent Information Leakage in Federated Learning shows that shared updates can leak latent information. We will report a simple attribute-inference probe on transmitted updates for our split versus FedAvg. We do not claim formal guarantees.

Baselines

The naive reference is purely local training. Each client trains the same recognizer on its own data with hindsight labels and never communicates. Our method must beat it on every client, and most clearly on the smallest ones, or federation adds nothing.

The strongest centralized reference is pooled training of the coupled intent-action model on all clients' data with human intent labels. For context, Coupling Intent and Action for Pedestrian Crossing Behavior Prediction reports 0.83 AUC on PIE intent, against 0.73 AUC for PIEint. Pie: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction reports 79% intent accuracy. Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction reports 79.28% accuracy on JAAD and 79.10% on STIP. These are upper references that use human labels and pooled data. We will not claim to beat them. Closing most of the gap to 0.83 AUC on PIE without labels or pooling would be a strong result. We also run a pooled hindsight-trained variant, which isolates the cost of federation from the cost of losing labels. We found no reported result on Lacuna for federated pedestrian intent estimation on PIE, JAAD or STIP.

The strongest federated competitors are four. The first is FedAvg on the full model, with the same hindsight objective, following the training recipe of the federated trajectory work above. The second is FedAvg followed by local fine-tuning of the output head, the standard and often very strong personalization recipe. The third is APFL's variance-scaled learning rates. APFL reports a CARLA prediction loss of 0.01160 versus 0.01760 for standard FL and 0.01643 for centralized cloud training, on driver trajectories rather than pedestrian intent. The fourth is the Ditto-style weight-drift penalty, plus its latent-space replacement from Adaptive Latent-Space Constraints in Personalized Federated Learning, which moved accuracy from 85.5% to 91.2% on its synthetic heterogeneity benchmark. We will use the cheaper multi-kernel variant if the deep-kernel variant is impractical at our scale.

Fairness rules apply to every method. All methods share the recognizer architecture, the hindsight labels, the number of communication rounds, the local epochs per round and a fixed hyperparameter search budget, the same number of configurations each, tuned on client-held-out validation splits. Beating a baseline means a higher mean intent AUC on human annotations and a higher worst-client AUC across three seeds, with no worse calibration error.

Two ablations isolate the key ingredient. The reverse split shares the recognizer and personalizes the outcome model. It tests whether the choice of half matters, rather than just the fact that a split exists. The second ablation removes ego-speed conditioning from the outcome model. It tests whether action conditioning is what turns hindsight labels into intent.

Why not just…?

Why not just run FedAvg and fine-tune the head locally? This is the most serious threat, because it is simple, personalizes, and uses the same local hindsight labels. Our argument is that it personalizes the part closest to the outcome and shares the cue-reading backbone. That is the wrong way round if cue meaning varies by city. ATPFL: Automatic Trajectory Prediction Model Design under Federated Learning Framework found plain FedAvg beat existing personalized methods for coordinate trajectory forecasting, so personalization is not automatically helpful. We see that result as consistent with our view: pure motion is largely shared. If FedAvg with head fine-tuning matches us on real data and in the cue-flip simulation, the claim is wrong and we will report it as such.

Why not just correct each client's base rate of crossing? Much of the cross-city difference might be how often people cross, not what cues mean. A per-client prior adjustment on top of FedAvg would fix that for almost no cost. We include this as a baseline. The local prior $p_{\psi_k}$ in our model is exactly this correction, so the extra value we claim must come from the personalized recognizer. The controlled simulation below, where base rates are held equal but cue meaning flips, is designed to settle the question.

Why not just pool anonymized data centrally? Blurring faces does not remove gait, location and routine. Beyond that, APFL found pooled cloud training worse than personalized federation on heterogeneous drivers (0.01643 versus 0.01160 loss). Pooling produces an average model. Our pooled hindsight variant measures this directly for pedestrians. If it wins clearly, the federated method's case rests on privacy alone, and we will say so.

Why not just use APFL's automatic variance rule instead of a hand-chosen split? APFL is attractive because it requires no decision about which half to personalize. Weight variance, however, says nothing about whether a parameter encodes cue meaning or outcome dynamics. In deep networks, small weight differences can map to large differences in what the representation means, which is the point made by the latent-constraints paper. We will inspect which layers APFL ends up personalizing. If it independently discovers our split and matches our performance, our contribution becomes an explanation of why it works rather than a better method.

Proposed Experiments

The real-data study treats datasets and cities as clients. PIE (Toronto) is one client and JAAD is another. STIP's footage from eight cities is split by city if city identifiers are recoverable, and by recording session otherwise. This gives roughly ten clients with natural heterogeneity in appearance, infrastructure and pedestrian behavior. Training uses only hindsight outcomes and ego telematics. Evaluation uses human intent annotations on held-out clips from every client.

We report intent AUC and accuracy, worst-client AUC, and expected calibration error. Calibration matters because confidence is reported to be poorly calibrated under distribution shift (Pie: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction). We also report anticipation curves of AUC versus time before crossing onset, since performance is known to peak about one second ahead (Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction). A leave-one-client-out test measures how fast a new city's recognizer adapts against a frozen shared outcome model. This is where we expect sharing the outcome model to help most.

The decisive check is a controlled CARLA study (CARLA: An Open Urban Driving Simulator). Scripted pedestrians carry a ground-truth intent, and towns act as clients, with two opposite regimes. In the cue-flip regime, the same visible cue, such as orienting toward the road, signals crossing in some towns and waiting in others. Intent-to-motion and base rates are identical across towns. In the motion-flip regime, cues mean the same thing everywhere, but committed crossers move differently, for example hesitating or walking fast. The mechanism predicts a specific pattern. Our split should win clearly in the cue-flip regime, the reverse split should win in the motion-flip regime, and base-rate correction should help in neither. Uniform gains for every personalized method would instead mean we are seeing generic personalization, not the proposed mechanism.

A closing closed-loop test feeds intent beliefs into a simple speed planner at pedestrian crossings. We measure time-to-pass and minimum distance, to check that better-calibrated, earlier intent estimates reduce hesitant stalls without closing safety margins.