Spending Frames Where Events Happen: Mean-Field Particle Sampling for Fast Video Events

Many video tasks are decided in one or two frames: the instant a racket meets a ball, the moment a gymnast leaves the beam. Models that look at every frame ("dense temporal sampling") do much better on these tasks than models that look at 8 or 16 evenly spaced frames. The catch is cost. A strong visual encoder run on every frame of a long rally or routine is expensive, so dense systems settle for a light encoder everywhere, and that light encoder struggles with the motion blur that makes these frames hard. We propose to keep a fixed budget of $K$ expensive frame evaluations and treat them as a cloud of interacting particles on the time axis. Each particle carries a position and a weight. Over a few rounds, particles drift toward moments the expensive encoder finds informative, push away from one another so that repeated events all get covered, and gain or lose weight, with low-weight particles respawned elsewhere. Because every particle reacts to the distribution of all the others rather than to each one individually, one update round costs $O(K)$ rather than $O(K^2)$. This is the mean-field idea, borrowed from the physics of large interacting populations.

The bet is concrete. At equal encoder compute, a closed-loop particle allocator, in which the expensive encoder's own outputs decide where to look next, should beat both uniform dense processing with a light encoder and a one-shot "cheap scorer picks the top-$K$ frames" cascade on fast, frequent, fine-grained event spotting. If the closed loop adds nothing over the one-shot cascade, the mean-field machinery is unnecessary, and the experiments are built to show that if it happens.

Core Claim

Dense sampling helps because the signal sits at a few transient moments, not because every frame matters equally. The evidence for density comes from SwinBERT's ablation on MSRVTT, where CIDEr rose from 36.6 at 2 frames to 55.3 at 64 frames (SwinBERT). Further evidence comes from FineGym's observation that denser sampling improves element recognition (FineGym) and from F³ED's reliance on every-frame processing to localize one-to-two-frame tennis shots (F³Set). If the signal really is concentrated in time, the right object to learn is an allocation density over time: a description of how much expensive computation each moment deserves. A finite set of weighted particles is the natural way to approximate such a density and refine it.

We define the allocation as the minimizer of a free energy over distributions $\mu$ on the clip's time interval $[0,T]$:

$$ \mathcal{F}(\mu) = \int V_\theta(t)\, d\mu(t) + \tfrac{1}{2}\iint W(t-s)\, d\mu(t)\, d\mu(s) + \varepsilon \int \mu \log \mu . $$

Here $V_\theta$ is a learned potential that is low where an event is likely. It is computed from a cheap all-frame stream and refined by the expensive features gathered so far. $W$ is a short-range repulsion kernel that stops particles from piling onto one shot when the next shot arrives a second later. $\varepsilon$ sets an entropy floor that keeps some exploration everywhere. The particle system approximates gradient descent on $\mathcal{F}$ using two kinds of move. The first is a transport move: positions follow $-\nabla V_\theta - \nabla W * \mu_K$, where $\mu_K$ is the current empirical particle measure. The second is a Fisher-Rao move, which multiplies each weight up or down according to its local excess free energy and respawns particles whose weight has collapsed.

Relation to Prior Work

The method is assembled from mean-field particle theory. Lazić et al. build particle systems whose particles carry evolving mass weights, used to discretize Fisher-Rao gradient flows in entropic optimization, and they prove that the empirical measure converges to the mean-field flow. We reuse their weighted-particle construction as the reweighting step. Their result is qualitative and assumes a compact state space; our time interval is compact, but we inherit the lack of rates. Priser et al. prove long-run convergence of discrete-time McKean-Vlasov particle systems (particles whose dynamics depend on their own population's distribution) as step size shrinks and particle count grows. This underwrites the claim that a few discrete rounds with $K$ particles approximate the limiting allocation.

Two papers constrain the design. Regularized Frank-Wolfe for Dense CRFs shows that plain mean-field updates in densely connected models can oscillate, and that regularized, damped steps converge. We therefore damp both moves rather than iterating fixed points naively. The self-attention CRF for group activity recognition (Empowering Relational Network) uses mean-field inference with dynamic halting and reports convergence in 3 to 5 iterations. We adopt halting on the change in $\mu_K$, and we take its iteration count as a prior for our round budget. That work also depends on detected actors and AlphaPose keypoints, and both FineGym and Go Beyond Earth document that pose estimation breaks under flips and microgravity. Our particles therefore live on raw-frame time, not on actor graphs.

Several dense-video papers motivate the problem. F³ED runs a TSM encoder (a temporal-shift CNN) on every frame and adds a hitting-moment localizer, a multi-label attribute head, and a bidirectional GRU. It is our host architecture and our strongest competitor. SwinBERT's learned sparse attention mask prunes token interactions but still encodes every frame and is limited to 64 frames. Temporal Query Networks use a stochastic feature bank, which makes dense training fit in memory but does not reduce inference encoder cost. None of these decide where to spend encoder compute. The mean-field analyses of transformers (Transformers Learn Nonlinear Features In Context, Infinite Limits of Multi-head Transformer Dynamics) treat tokens as particles in order to analyze training. We instead use particles as a computational device that chooses which frames to compute at all.

Research Program

The system has three parts.

The first is a cheap stream: a small, low-resolution TSM run on every frame. It costs a small fraction of the expensive encoder and produces an initial potential $V^{(0)}$.

The second is the particle allocator. It runs $S$ rounds, with $S$ between 3 and 5 by the halting criterion. In each round the expensive encoder (a full-resolution video backbone over a short window around each particle) is evaluated at the $K$ particle positions. Its outputs update $V_\theta$ locally through a small network. Particles then take one damped transport step and one Fisher-Rao reweighting step. The repulsion term $W * \mu_K$ is evaluated by kernel smoothing on a frame grid, which is linear in $K$ and $T$.

The third part is F³ED's heads. They read expensive features at the final particle locations, interpolate them onto the frame grid, fuse them with cheap features elsewhere, and output per-frame event and attribute predictions.

Particle positions are discrete choices, so we do not backpropagate through them. Training uses two losses. The detection heads are trained with F³ED's losses. The potential $V_\theta$ is trained so that the stationary density of the dynamics, which is proportional to $\exp(-V_\theta/\varepsilon)$ when there is no interaction, matches a target density. The target is ground-truth event times smoothed with a narrow Gaussian and mixed with a uniform floor. A KL loss on the frame grid enforces the match. This supervision is simple and avoids reinforcement learning. The cost is that it encodes our prior about where compute should go, so we also test a variant that weights the target by per-frame detection loss.

The analysis has two parts. The first is a budget-elasticity study. If the particle measure approximates a limiting density, a model trained at budget $K$ should transfer to $K/2$ and $2K$ at test time with smooth accuracy changes. The second is a check of whether $\mu_K$ actually stabilizes across rounds, which tests the damping that the Frank-Wolfe result motivates.

Baselines

The naive reference point is uniform sampling at the same budget: $K$ evenly spaced expensive evaluations plus the cheap stream, feeding identical heads. This tells us whether allocation matters at all. Above it sits uniform dense expensive processing, the same heads with the expensive encoder on every frame. This is the accuracy ceiling at many times our compute, and it defines how much of the gap we close.

The strongest existing method is F³ED itself (F³Set), which reports 40.3% F1 and an Edit Score of 74.0 on the finest tennis granularity $G_{high}$, against 31.4% F1 for the best of SlowFast, ActionFormer, and E2E-Spot. We will retrain F³ED from its published design on the same splits, report our reproduction alongside 40.3, and measure its encoder FLOPs ourselves, since the file gives none. Beating it means higher $G_{high}$ F1 at equal or lower measured encoder FLOPs, or equal F1 at substantially lower FLOPs.

The most important competitor for the specific claim is a cascade. It uses the same cheap stream and the same expensive encoder, selects the top-$K$ frames by cheap-stream event score with temporal non-maximum suppression (discarding near-duplicate peaks), and makes one pass. This is our method with no rounds, no interaction among particles, and no feedback from the expensive encoder. It shares our encoder, data, training schedule, and hyperparameter search budget.

Three ablations isolate the ingredients: removing the repulsion kernel $W$; transport only, with no reweighting or respawn; and reweighting only, with fixed positions. All runs use identical backbones, an equal number of tuning trials, and three seeds, with F1 differences reported against seed variance.

Why not just…?

Why not just run the cheap scorer once and take its top-$K$ frames? This is the real threat. If the cheap stream already finds the events, closed-loop refinement is ornamental. The case for the loop is that the cheap stream is weakest exactly where blur is worst, and that expensive features carry structure the cheap stream lacks: a recognized serve predicts a return roughly a second later, and that prediction can pull particles there. The open-loop-versus-closed-loop comparison at equal FLOPs settles the question. If the gap is within seed noise on $G_{high}$, we will report that mean-field dynamics are unnecessary and that the cascade suffices.

Why not just run a light encoder densely, as F³ED does? F³ED shows this is a strong recipe, but its 40.3% $G_{high}$ F1 leaves most fine-grained events wrong. The Diving48-v2 gap between TSM (51.1%) and TQN (81.8%) reported by Temporal Query Networks suggests that light temporal encoders lose fine detail. The uniform dense-expensive ceiling tells us whether a heavier encoder at critical frames helps at all. If even dense expensive processing barely beats F³ED, allocation cannot help, and we will say so.

Why not just use learned sparse attention or a feature bank? SwinBERT's mask prunes attention after every frame has already been encoded, and is capped at 64 frames. TQN's bank saves training memory, not inference compute. Neither reduces the per-frame encoder cost that dominates on long untrimmed video, so neither is a like-for-like compute baseline. Instead, we report encoder FLOPs per minute of video for all methods.

Decisive Experiments

The primary benchmark is F³Set tennis at $G_{high}$, with badminton and table tennis as transfer checks. We report F1 and Edit Score against encoder FLOPs across $K$, plotting accuracy-compute curves for our method, the cascade, uniform-$K$, F³ED, and the dense-expensive ceiling. The decisive result is the closed-loop curve lying strictly above the cascade curve at matched FLOPs. A secondary result is that the repulsion ablation hurts most on rallies with the shortest gaps between shots, which is where repulsion should matter.

On FineGym we use element-level accuracy on Gym99 against TQN's reported 93.8%, and we run long-routine inputs where uniform budgets thin out. The prediction is that our advantage over uniform-$K$ grows with routine length.

Diving48-v2 is a negative control. Every frame of a dive carries rotation information, so the target density should be nearly flat, and we predict no gain over uniform-$K$ against TQN's 81.8%. A gain there would suggest the method is acting as generic test-time augmentation rather than doing allocation.

Finally, the budget-elasticity test asks whether a model trained at $K$ degrades gracefully at $K/2$ and $2K$, where the cascade would need retuning of its suppression window. This is the property the particle-measure view predicts and the one-shot alternative does not.