Abstract
Recent works have shown that deep learning (DL) models can effectively learn city-wide crowd-flow patterns, which can be used for more effective urban planning and smart city management. However, DL models have been known to perform poorly on inconspicuous adversarial perturbations. Although many works have studied these adversarial perturbations in general, the adversarial vulnerabilities of deep crowd-flow prediction models in particular have remained largely unexplored. In this paper, we perform a rigorous analysis of the adversarial vulnerabilities of DL-based crowd-flow prediction models under multiple threat settings, making three-fold contributions. (1) We propose CaV-detect by formally identifying two novel properties - Consistency and Validity - of the crowd-flow prediction inputs that enable the detection of standard adversarial inputs with 0% false acceptance rate (FAR). (2) We leverage universal adversarial perturbations and an adaptive adversarial loss to present adaptive adversarial attacks to evade CaV-detect defense. (3) We propose CVPR, a Consistent, Valid and Physically-Realizable adversarial attack, that explicitly inducts the consistency and validity priors in the perturbation generation mechanism. We find out that although the crowd-flow models are vulnerable to adversarial perturbations, it is extremely challenging to simulate these perturbations in physical settings, notably when CaV-detect is in place. We also show that CVPR attack considerably outperforms the adaptively modified standard attacks in FAR and adversarial loss metrics. We conclude with useful insights emerging from our work and highlight promising future research directions.
Introduction
Deep Learning (DL) models are increasingly central to smart city infrastructure, particularly for predicting city-wide crowd-flow patterns. By analyzing inflow and outflow data, these models assist in urban planning and public safety management. However, like many DL applications, crowd-flow models are vulnerable to adversarial attacks—subtle, intentional perturbations to input data that lead to significant prediction errors.
A critical gap in current research is that most adversarial attacks are "digital-only." They generate noise that is mathematically effective but physically impossible, such as negative crowd counts or violations of flow conservation. This paper addresses this gap by formalizing the physical constraints of crowd data and introducing a new, physically-realizable attack called CVPR.
The CaV-detect Defense
To identify adversarial inputs, the authors formalize two "priors" that any legitimate crowd-flow data must satisfy:
- Consistency: The historical data reported at time $t$ must match what was reported at time $t-1$. Standard attacks often create self-contradictory histories.
- Validity: This is a conservation law. The number of people entering a specific grid cell cannot exceed the total number of people who left the adjacent cells.
The proposed CaV-detect mechanism uses 2D convolution filters to check these properties in real-time. If an input violates these rules, it is flagged as adversarial and rejected.
The CVPR Attack: Bypassing Physical Checks
To demonstrate that simple physical checks are insufficient, the authors designed the CVPR (Consistent, Valid, and Physically-Realizable) attack. Unlike standard methods like PGD or FGSM, CVPR explicitly incorporates physical priors into its perturbation generation:
- Universal Perturbations: It applies consistent perturbations across the time history to ensure it passes consistency checks.
- Distribution Matrix ($W$): It uses a learned weight matrix to ensure that for every "inflow" perturbation added to a cell, a corresponding "outflow" perturbation is added to a neighbor, maintaining flow conservation.
- Physical Realizability: It constrains perturbations to be positive, reflecting the reality that an adversary can add people or devices to a crowd but cannot easily "delete" them from a physical sensor's view.
Evaluation and Results
The study utilized the TaxiBJ dataset, a benchmark containing Beijing taxi trajectories divided into a $32 \times 32$ grid. The experiments revealed a significant "arms race" between defense and attack:
- Detection Efficacy: CaV-detect achieved a 0% False Acceptance Rate (FAR) against standard digital attacks, successfully filtering out all physically impossible noise.
- Attack Efficacy: The CVPR attack bypassed CaV-detect with nearly 100% FAR, proving that sophisticated attacks can mimic physical laws to remain undetected while still degrading model accuracy.
- Model Robustness: Among tested architectures, the Temporal Graph Convolutional Network (TGCN) proved to be the most robust against these physically-realizable attacks.
- Limited Expressiveness: A qualitative finding showed that these models have "limited expressiveness." Even with significant control over the input, an attacker often couldn't force the model to predict an arbitrary state, as the models are heavily biased toward patterns found in the training data.
Conclusion and Future Work
While the CVPR attack exposes critical vulnerabilities, the authors acknowledge that executing such an attack in the real world remains a massive logistical challenge, requiring the precise coordination of thousands of mobile devices. Nonetheless, this research underscores the need for more robust, physics-aware defenses in smart city infrastructure. Future work will explore how vulnerabilities change with adversarial training or different city grid resolutions.
Summary updated