dKFD: Phase-Structured Evidence Allocation for Fixed-Budget Localized Event Understanding

Aditya Bagri, Ashutosh Kumar, Chaitanya Lakhchaura, Avinash Anand, Zhengkui Wang, Rajiv Ratn ShaharXiv 2026 · arXiv

TL;DR

dKFD introduces a phase-structured approach to sparse video selection, moving away from "global competition" where all frames compete for a single budget. By explicitly reserving capacity for pre-event context, the event itself, and post-event consequences, dKFD produces far more stable and explanatory evidence packs, improving selector-event alignment by over 30% on the DoTA anomaly recognition benchmark.

What Problem Does This Paper Address?

In sparse video understanding, the goal is to select a tiny subset of frames (a "fixed budget") to represent a much longer sequence. Most current methods use a Global Top-K strategy: they score every frame for saliency or importance and pick the top $K$.

While this works for general video summarization, the authors argue it is a poor inductive bias for temporally localized events (like a car accident). A Global Top-K selector often clusters all its picks at the "peak" of the event because those frames are the most visually discriminative. However, to truly understand or inspect an event, you need the causal setup (pre-event) and the aftermath (post-event). The authors show that Global Top-K methods produce highly unstable evidence that often misses this crucial context, even if they manage to classify the event correctly.

Methods

The authors propose dKFD (Phase-Structured Differentiable Selector). The architecture follows a specific pipeline to ensure selected frames serve as a coherent "evidence pack":

  1. Full-Sequence Temporal Encoding: The video is first processed by a frozen CLIP vision encoder, then passed through a Transformer-based temporal encoder. This allows each frame to "know" its relative position and role in the sequence.
  2. Phase-Aware Selection: Instead of one selector, dKFD uses three phase-specific selectors. It partitions the total budget $K$ into $K_{pre}$, $K_{evt}$, and $K_{post}$.
  3. Differentiable Masking: To keep the system end-to-end trainable, the model produces soft selector maps and hard Top-K masks, using a straight-through estimator to route gradients.
  4. Evidence-Conditioned Recognition: The final classification is "bottlenecked"—the model can only see the selected frames to make its prediction.
  5. Selector-Aware Grounding: While recognition is sparse, temporal grounding (predicting the start/end of the event) uses the full sequence but is conditioned on the selector's choices.
Fig. 1: dKFD overview. The full video is temporally encoded before phase-aware se- lectors allocate a fixed evidence budget across pre-event, event, and post-event frames. The selected evidence pack is used for classification, while selector maps and pooled evidence condition temporal grounding over the full timeline.

Datasets

  • DoTA (Detection of Traffic Anomalies): The primary benchmark used for controlled experiments, providing event labels and temporal intervals.
  • VRU-Accident: Used as a diagnostic benchmark for downstream Vision-Language Model (VLM) tasks like Video Question Answering (VQA) and dense captioning.
  • DAD (Dashcam Accident Dataset): Used to test the transferability of the learned selectors to accident anticipation tasks.

Key Insights and Ideas

  • Recognition $\neq$ Evidence Quality: A major "aha moment" is that a model can achieve high classification accuracy while picking terrible frames for human inspection. Global Top-K selectors are "unstable"—their frame selection varies wildly across different training seeds even when their accuracy remains high.
  • Phase Supervision is "Load-Bearing": Simply splitting the budget into three buckets isn't enough. The authors found that without explicit (weak) supervision to align selectors with pre/event/post phases, the model's performance collapses. The architecture provides the capacity for structure, but supervision provides the alignment.
  • Separable Roles: Ablation studies revealed a clear division of labor: temporal encoding is critical for accurate grounding (localization), while phase-aware selection is the primary driver of evidence alignment.
  • Task-Specific Utility: dKFD is an "event-centric" tool. In tasks like dense captioning that reward broad scene coverage, simple uniform sampling remains a tough baseline to beat.

Key Results

The most striking result is the improvement in Frame AUC, which measures how well the selected frames align with the actual event phases.

  • Alignment Gain: At a budget of $K=12$, dKFD improved Frame AUC by +30.97 points over a matched Global Top-K selector.
  • Stability: While Global Top-K results fluctuated wildly across seeds (straddling chance levels of alignment), dKFD remained tightly clustered and stable.
  • Downstream VQA: When dKFD's selected frames were fed into off-the-shelf VLMs (like Qwen2.5-VL), they consistently outperformed Global Top-K frames on accident-related questions (e.g., "What caused the accident?").
Table 1: Positioning of dKFD relative to neighboring video-selection and long-video understanding settings. dKFD studies supervised fixed-budget evidence allocation for temporally localized events.

Limitations and Future Work

  • Fixed Budgets: Currently, dKFD uses a hard-coded split (e.g., 3 frames for pre-event, 5 for event, 4 for post-event). Real-world events vary; some have long buildups and abrupt endings. Future work could explore adaptive or class-conditional budgets.
  • Domain Specificity: The study focused heavily on driving accidents. The "Phase-Structured Hypothesis" needs testing in other domains like sports or industrial monitoring.
  • Supervision Requirements: dKFD requires event-interval annotations for training. Developing semi-supervised or self-supervised versions of phase-alignment would make the method more scalable.

Summary updated

Abstract

Sparse video understanding often requires selecting a small set of visual evidence under a fixed frame budget. Most sparse selectors allocate this budget globally, allowing all frames to compete with one another. For temporally localized events, this can be a poor inductive bias: useful evidence is often distributed across pre-event context, the event itself, and post-event consequences. We study fixed-budget evidence allocation for localized event videos and show that globally competitive Top-$K$ selectors can preserve recognition and grounding while producing unstable event evidence. On DoTA Video Anomaly Recognition, Global Top-$K$ obtains competitive recognition and temporal grounding, but low selector-event alignment at $K=12$ (Frame AUC $51.5 \pm 10.8$). We propose dKFD, a phase-structured differentiable selector that reserves evidence capacity across pre-event, event, and post-event phases after full-sequence temporal encoding. Under matched-budget multi-seed evaluation, dKFD improves Frame AUC by $+30.97$ over a matched Global Top-$K$ selector at $K=12$ ($p<0.01$), while yielding modest but statistically significant recognition gains and comparable temporal grounding. Mechanism ablations show that phase supervision is load-bearing: removing it reduces Frame AUC to $41.1 \pm 12.0$ even when phase-partitioned budgets are retained. Downstream diagnostics on VRU-Accident show consistent gains over learned Global Top-$K$ across VLM families, while dense captioning reveals a boundary condition where uniform sampling remains competitive. These results support phase-structured allocation as a controlled fixed-budget approach for event-centric sparse evidence selection, not as a universal video summarization strategy.
BibTeX
@misc{bagri2026dkfd,
  author = {Aditya Bagri and Ashutosh Kumar and Chaitanya Lakhchaura and Avinash Anand and Zhengkui Wang and Rajiv Ratn Shah},
  title = {{dKFD: Phase-Structured Evidence Allocation for Fixed-Budget Localized Event Understanding}},
  year = {2026},
  eprint = {2609.33083},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.33083},
}
Figure