Nowcasting the Slow Branch in Streaming Video Surveillance

Video surveillance systems face a choice that public health surveillance has dealt with for decades. A fast, cheap video model can raise an alert within milliseconds but makes many mistakes. A slow, accurate model, such as a large video-language model or a full-resolution anomaly detector, reaches a better verdict, but only seconds after the event. Today's streaming systems either wait for the slow model or ignore it. Epidemiologists in the same position, holding official case counts that arrive one to three weeks late, do neither. They nowcast: they estimate today's unreported value by combining the most recent late reports with timely but noisy proxies such as search queries. We propose to treat the slow model's verdicts in a decoupled video pipeline as a lagging report stream and to nowcast them.

Concretely, a small model receives two inputs: the stale verdicts the slow model has already finished, and the fast model's features for the frames it has not yet reached. From these it predicts what the slow model will say about the present moment, with an uncertainty estimate that narrows as the real verdict approaches. The same uncertainty decides which frames and image tiles the slow model should process next, much as public health systems direct scarce lab testing to the most informative sites. The goal is to give alerts close to the fast model's latency and close to the slow model's accuracy, and to measure this as event-to-alert delay rather than throughput in frames per second.

Core Claim

In an asynchronous video pipeline, the slow model's lag is a reporting delay with a known structure, not just a nuisance. A lag-aware estimator that conditions on delayed slow verdicts should beat both components and a confidence-based cascade on detection delay at a fixed false-alarm rate. Two facts support this. The slow model's past verdicts carry scene-specific information that the fast model cannot recover from the current frame. The fast model's current features carry timing information that the slow model's backlog cannot provide.

Let $y_s(t)$ be the slow model's anomaly score for the window ending at time $t$, available only at time $t + D_t$, where $D_t$ is its processing delay. Let $z(t)$ be the fast branch's feature vector, available almost immediately. The nowcaster estimates the unseen slow verdict from the verdicts already reported and from the fast features since then:

$$\hat{y}_s(t) = f\big(\{y_s(\tau) : \tau + D_\tau \le t\},\; z(t-k{:}t)\big).$$

The first version of $f$ deliberately copies ARGO. ARGO combines autoregression on lagged official influenza reports with a sparse, L1-regularized regression on search-query volumes, refit over a rolling window. Our analogue is an autoregression on the reported slow scores plus a sparse regression on the fast features. It is refit online per camera and reports quantile outputs for calibration. A small recurrent version comes second, and only if the linear one leaves measurable error on the table.

Where This Sits in Prior Work

The decoupling itself comes from streaming video-language models, and these papers supply the method. VideoLLM-online runs a fast image encoder alongside a slower language decoder with a rolling memory of past computation, and reaches 10–15 frames per second (FPS) on an A100. Dispider splits perception into a "Watcher" that decides whether to respond and a larger "Speaker" that responds on demand. Both treat the gap between perception and reaction as something to shrink through engineering or bridge with a learned "respond now / stay silent" gate. Neither estimates what the slow component will conclude before it finishes. That estimate is the piece we add.

On the surveillance-video side, FusedVision runs two branches at once, a reconstruction autoencoder and a YOLOv8 object detector. It reaches 47 FPS and improves area under the curve (AUC) by 5.45% on ShanghaiTech over a standalone reconstruction baseline. It fuses branches that run at the same rate. We fuse branches that run at different rates and with variable lag, so FusedVision serves as our strongest fast-path baseline rather than a competitor in kind.

Codec work motivates the scheduling half. HEVC tile- and frame-level parallel encoding splits frames into independently encodable tiles. Wavefront row scheduling starts a dependent row as soon as the context it needs is ready. Horizontal frame slicing across media processors scales throughput linearly with the number of processors. All three schedule by fixed dependency order. We schedule the slow model's tiles by where the nowcast is least certain.

The surveillance-latency literature constrains the design in two ways. First, the Korean rhinovirus search-trend study and the U.S. state-level early warning system both stress that proxy models selected with access to the whole history overstate how well they will work prospectively. We therefore fit and evaluate strictly online, never selecting features with test-period data. Second, the history of Google Flu Trends, as recounted in the ARGO work, shows that proxies drift and must be recalibrated against the late reports as those reports arrive. This is why the nowcaster keeps refitting against actual slow verdicts instead of being trained once.

Research Program

The work proceeds in three parts.

Measure the lag. We first characterize the lag in realistic decoupled pipelines. We record the full distribution of $D_t$, not just its mean, for a slow branch under burst load, and check whether it depends on scene content. Content-dependent lag would mean the hardest events wait longest.

Build the nowcaster. The second part is the nowcaster described above, with a rule for issuing provisional alerts when the lower quantile of $\hat{y}_s(t)$ crosses the alarm threshold. Each provisional alert is confirmed or retracted when the true $y_s(t)$ arrives.

Schedule by uncertainty. The third part is the dispatcher. It splits each frame into tiles in the HEVC style and sends the slow model the tiles and windows with the widest nowcast interval, subject to a compute budget. This is the ingredient most likely to matter when the slow model cannot keep up with every camera.

Main testbeds. The main testbeds are ShanghaiTech, Avenue, and UCSD Ped2, the benchmarks FusedVision used.

  • Fast branch: a downsampled detector plus autoencoder.
  • Slow branch: two variants. One is a full-resolution, all-tile version of the same two-branch network. The other is a video-language model queried per window in the VideoLLM-online style.
  • Controlled lag: we inject synthetic delays of 0.5 to 10 seconds, alongside the measured lag, so that the gain can be plotted against delay.

Secondary testbed. ETBench streaming temporal grounding tests whether the idea carries over to streaming language models. It asks a model to locate, while the video plays, when a described event happens. Here the Speaker plays the slow branch and the Watcher's features play the fast proxy.

Clinical check. A short out-of-domain check uses MC-MED. Its 1-minute vital signs act as the fast proxy and its laboratory results act as the delayed reports. This tests whether nowcasting late labs recovers the value that Ueno and colleagues found delayed labs fail to add for predicting cardiac arrest. MC-MED has no video, so this checks the estimator, not the vision system. We set aside the ED video cohorts (hospitalization prediction from triage clips, respiratory rate from ED video). With 723 and 19 patients and offline clips, they cannot support a streaming latency evaluation.

Baselines

Do-nothing reference. The slow branch alone, with the alert time set to when its verdict actually arrives. This is what a cautious deployment does now. It fixes the accuracy ceiling and the latency floor we try to escape.

Fast branch alone. The FusedVision-style network (YOLOv8 plus autoencoder) at reduced resolution. FusedVision reports 47 FPS and a 5.45% AUC gain on ShanghaiTech. We retrain it on the same splits and hardware with the same tuning budget, since our fast branch differs in resolution.

Last-value carry-forward. Repeat the most recent slow verdict until a new one arrives. This mirrors relying on the latest official report and isolates the value of the fast features.

Confidence cascade. Trust the fast model, and escalate to the slow model only when the fast model is uncertain. We expect this to be the strongest simple competitor. We tune its threshold on the same validation data and with the same search budget as the nowcaster.

Dispider-style gate (ETBench only). A Dispider-style decision gate, which reaches F1 36.1 on ETBench streaming temporal grounding versus 13.2 for VideoLLM-online according to Dispider. Beating it means higher F1 at equal or lower mean response delay.

Ablations. Two ablations isolate the key ingredient. Removing the fast features leaves pure autoregression on slow verdicts. Removing the slow history leaves an online-recalibrated mapping of fast features alone. Replacing uncertainty-driven dispatch with wavefront-order or uniform tile scheduling isolates the dispatcher.

All methods share one compute budget, defined as slow-branch tile-seconds per wall-clock second.

Success criterion. The proposal succeeds if it achieves lower median event-to-alert delay than the cascade at matched false-alarm rate and frame-level AUC within one point of the slow branch. The gain should also grow with injected delay.

Why not just…?

Why not just distill the slow model into the fast one? This is the most serious alternative. A fast model trained to imitate the slow one removes the lag entirely if imitation succeeds. The nowcaster's advantage has to come from information distillation cannot use: the actual slow verdicts on this camera's recent past, which encode scene-specific normality. We will train a distilled fast branch at matched compute and include it as a baseline. If it matches the nowcaster on alert delay at fixed false-alarm rate across all three surveillance datasets, the claim is false and we will say so.

Why not just use a confidence-thresholded cascade? A cascade decides whether to wait. It never estimates what the slow model will say, so every escalated event pays the full lag $D_t$. The nowcaster issues a provisional alert during that lag. The gap between the two should therefore widen as injected delay grows. If the gap stays flat, the cascade is sufficient.

Why not just parallelize the slow model until the lag disappears? Tile, frame, and wavefront parallelism (HEVC tiles, frame slicing) do cut latency, and with unlimited hardware the problem goes away. This is an honest boundary condition. We will report how much lag remains under realistic multi-camera load at fixed hardware. A language-model slow branch, even at VideoLLM-online's 10–15 FPS on an A100, cannot cover many feeds at once. If tile parallelism alone pushes $D_t$ below one frame interval in a regime, we will mark that regime as one where the method is unnecessary.

Decisive Checks

Delay curves. The first check plots median and 95th-percentile event-to-alert delay against false-alarm rate on ShanghaiTech, Avenue, and Ped2, for every baseline. The only published reference point in our file is FusedVision's 5.45% AUC gain at 47 FPS on ShanghaiTech. We found no reported result on Lacuna for Avenue or Ped2.

Calibration. The second check asks whether provisional-alert intervals cover the eventual slow verdict at their nominal rate. Retraction rates must stay low enough to be usable, since an alert that flips often is not an alert.

Gain versus lag. The third check is the gain-versus-delay curve under injected lag. Our account predicts a curve that rises and then plateaus once the fast features carry no more information about the window.

Prospective validity. Every number is computed online, with no hyperparameter or feature choices made on test-period data. This is the lesson the search-trend literature learned at some cost.