Action detectors that work on categories they never saw in training now exist, but only in hindsight: they look at a finished video and mark where an unseen action such as "pole vault" started and ended. Systems that predict actions before they happen exist too, but only for a fixed list of trained categories. When those predictors run on a continuous, unedited stream, their accuracy collapses as the forecast reaches further ahead. This note proposes a model that does both: from a live stream it predicts that an action described only by a text prompt will start, and how many seconds from now. The core bet is that much of the collapse is not a failure to see the future. It is a failure to calibrate. The scores a vision-language model gives unseen categories shift in level and spread as the lead time grows, so a single fixed decision rule produces floods of false alarms far ahead and misses close in.
The proposal borrows two ideas from outside video. Time-series forecasting has shown that the statistics of a future window drift away from those of the observed window, and that predicting the future window's statistics separately reduces long-range error. Language processing has shown that relative time phrases ("in two days") should be resolved against an explicit anchor rather than treated as free-floating values. We apply the first idea to the score space of an open-vocabulary detector, and the second to how start times are encoded. The result is a zero-shot anticipator whose start-time predictions are offsets from "now" and whose class scores are re-centered for each lead time.
Core Claim
The claim has two parts, and the second is the one that could fail. First, in zero-shot untrimmed anticipation, the distribution of text-video similarity scores for unseen classes changes with lead time. Far-ahead scores are flatter and are dominated by scene and object cues rather than motion. This mismatch, rather than the visual representation itself, accounts for a large share of the false alarms and of the accuracy decay. Second, the correction must depend on the current scene, not only on the lead time. That is, the future window's score location and scale should be predicted from what is currently observed, as Dish-TS predicts horizon-window mean and variance from the lookback window for numeric series. A fixed per-lead-time temperature fitted on seen classes should not be enough. If a fixed per-lead-time temperature matches the scene-conditioned version, the second claim is false. The contribution then shrinks to a calibration finding, and we will report it that way.
What the prior work gives us and leaves open
Three lines of work motivate the gap. On the retrospective side, Ti-FAD injects CLIP text embeddings into every layer of an ActionFormer-style detector. CLIP is an image-text model trained to match pictures with captions. Ti-FAD also masks attention around likely action peaks to avoid confusing shared sub-actions, and it reaches 41.2% average mAP on zero-shot THUMOS14. GRIZAL enriches thin class prompts with generated text and images. Both methods only localize actions that have already been fully observed, and we reuse Ti-FAD's text-infused attention as our encoder, restricted to past frames.
On the predictive side, Untrimmed Action Anticipation defines the task we adopt: at every 0.25 s of an EPIC-KITCHENS-100 stream, predict (class, time-to-start) pairs for actions starting in the next 5 s. It reports closed-vocabulary action mAP of 0.84% at a 1 s tolerance, and false alarms on 98% of timestamps with no upcoming action. It also found that a continuous time-to-start regressor failed where discrete per-second classifiers succeeded. That finding constrains our design: we encode start time as a discrete lead-time bin plus a small within-bin residual. Both are measured from the current-time anchor, so the start time is an offset from "now", which is the anchor-plus-offset form used in temporal normalization (Bethard). The bin-plus-residual encoding also settles the regression-versus-bins tension the literature leaves open. Video + CLIP Baseline for Ego4D LTA shows that CLIP features help long-term anticipation of objects more than of verbs, a "temporal blindness" that is our reason to expect lead-time-dependent score drift. It predicts action sequences rather than start times.
On normalization, Dish-TS supplies the method. One network normalizes the observed window, another predicts the future window's location and scale, and this beat RevIN by over 10% on average. RevIN is the simpler alternative that normalizes the observed window and reuses its statistics. Time-Conditioned Action Anticipation in One Shot supplies the practice of feeding lead time directly into the network rather than rolling forward step by step, which reached 50.0% versus 38.1% for an iterative CNN at 10 s on 50Salads. SimpleMem is evidence from a different domain that anchoring relative times matters: removing ISO-timestamp normalization cut long-horizon temporal reasoning by 56.7%. None of these handle unseen classes or video start times.
Research Program
The model encodes the observed past with a causal version of Ti-FAD's text-infused encoder. For each lead-time bin $k$ (twenty 0.25 s bins up to 5 s), it produces a future-query embedding $v_k$ by conditioning on an embedding of the bin's lead time, in the one-shot style rather than by rollout. Each class prompt $c$ is embedded as $e_c$. A small horizon-coefficient network $h$ sees only class-agnostic inputs: pooled observed features $x$ and the lead time $\tau_k$. It outputs a location $\mu_k(x)$ and scale $\sigma_k(x)$. The onset score is
$$s_{k,c} = a_k(x) + \frac{\cos(v_k, e_c) - \mu_k(x)}{\sigma_k(x)},$$
where $a_k(x)$ is a class-agnostic "something starts in bin $k$" logit, trained on all seen-class onsets. Because neither $a_k$ nor $h$ ever sees class identity, both should transfer to unseen prompts. A residual head refines the start time within the predicted bin.
Training uses seen classes only. The loss is a per-bin focal classification loss over the joint score and an $L_1$ loss on the residual. We will also analyze how the empirical mean and spread of $\cos(v_k, e_c)$ on held-out classes change with $\tau_k$. That analysis is the direct test of the first claim.
Baselines
The naive reference is "copy the present." A causal Ti-FAD detects which unseen action is currently happening and predicts that it, or the class with the highest current similarity, starts at a fixed prior offset equal to the training-set median lead time. This baseline shows how much the task rewards actual forecasting.
The strongest existing competitor is the discrete-bin anticipator of Untrimmed Action Anticipation, made open-vocabulary by replacing its class weights with CLIP text embeddings of the class prompts. Its only reported number is closed-vocabulary: action mAP of 0.84% at 1 s tolerance on EPIC-KITCHENS-100, with 98% false alarms on no-action timestamps. There is no reported result on Lacuna for zero-shot untrimmed anticipation. We will run this text-headed version ourselves on the same features, splits, training budget, and hyperparameter search as our model. Beating it means higher unseen-class mAP at the 0.5 s and 1 s tolerances, a lower false-alarm rate on no-action timestamps, and a flatter mAP-versus-lead-time curve, all across three seeds.
The ablations isolate the key ingredient. One removes $h$ entirely. A second replaces $h$ with one fixed temperature and offset per lead-time bin, fitted on seen-class validation data. This is the critical ablation for the second claim. A third replaces $h$ with test-time z-scoring of similarities across the prompt vocabulary at each timestamp. A fourth swaps bin-plus-residual timing for direct continuous regression, to check whether the regression failure reported by Untrimmed Action Anticipation recurs in the zero-shot setting.
Why not just…?
Why not just cascade a video predictor with Ti-FAD? Generating future frames and running a retrospective detector on them is slow: STRPM takes about 4 s per 4K frame. Mean-squared-error predictors also blur exactly the motion detail that separates verbs. We will still run a low-resolution feature-space version, in which future features are predicted autoregressively and then detected. That version tests whether forecasting in feature space with a fixed decision rule already solves the problem.
Why not just prompt a video language model for future timestamps? Pure neural generation is fragile on time arithmetic, which is why temporal normalization pairs neural models with symbolic resolution (Rigau et al.). Free-text outputs also give no calibrated per-bin scores for computing mAP. We will include a prompted model on a subsample as a reference. We expect it to fail mainly on timing precision and false alarms, and we will measure both rather than assume it.
Why not just fit one temperature per lead time? This is the real threat. If score drift depends only on lead time, a twenty-parameter calibration fitted on seen classes recovers most of the gain at no cost. That is the second ablation above. If it matches the scene-conditioned network on unseen classes, we will drop the claim that $h$ must see context. The method then becomes "lead-time calibration plus anchored bins," which is still useful but smaller.
Why not just z-score across the vocabulary at test time? That approach normalizes over classes, not over the future window, and it depends on which prompts happen to be in the vocabulary. It is the third ablation, and we will additionally test it with vocabularies padded by distractor prompts.
Proposed Experiments
The main benchmark is the EPIC-KITCHENS-100 untrimmed anticipation protocol with zero-shot splits we define by holding out verb classes. Holding out verbs is the hard case given CLIP's verb weakness, and we will also hold out whole verb-noun actions. The second benchmark is THUMOS14 recast as anticipation, using the standard 50/50 and 75/25 zero-shot splits, sampling every 0.25 s and asking for onsets within 5 s. Ti-FAD's 41.2% retrospective mAP serves as a rough ceiling on what hindsight achieves there. Metrics are anticipation mAP at tolerances of 0.25, 0.5, 1 and 2 s, false-alarm rate on no-action timestamps, the slope of mAP against lead time, and expected calibration error per lead-time bin on unseen classes.
The decisive checks are three. First, the measured mean and spread of unseen-class similarities must actually change with lead time; if they do not, the premise fails. Second, the scene-conditioned $h$ must beat the per-lead-time temperature on unseen classes, not just seen ones. Third, the gain must concentrate in far bins and in no-action false alarms, where the drift hypothesis predicts it should be largest. A secondary check runs on Ego4D long-term anticipation. There we will see whether lead-time normalization moves verb edit distance below the 0.7125 reported for the Video + CLIP baseline, which tests whether the effect carries over to sequence-level forecasting.
References
- Dish-TS: A General Paradigm for Alleviating Distribution Shift in Time Series Forecasting
- Text-Infused Attention and Foreground-Aware Modeling for Zero-Shot Temporal Action Detection
- GRIZAL: Generative Prior-guided Zero-Shot Temporal Action Localization
- Untrimmed Action Anticipation
- A Synchronous Context Free Grammar for Time Normalization
- Video + CLIP Baseline for Ego4D Long-term Action Anticipation
- Time-Conditioned Action Anticipation in One Shot.
- SimpleMem: Efficient Lifelong Memory for LLM Agents
- STRPM: A Spatiotemporal Residual Predictive Model for High-Resolution Video Prediction
- A modular approach for multilingual timex detection and normalization using deep learning and grammar-based methods