Localizing Brief Actions within Long Untrimmed Videos

This direction is split into other directions: temporal action sparsity. Its last version stays readable and citable here.

Version 1 of 165 papers66 conceptsfrozen 2026-10-04

In the early days of computer vision, video understanding focused on "trimmed" clips—short, curated snippets where the action (like "brushing teeth") filled the entire duration. But real-world video is messy. Whether it is hours of surveillance footage, a full-length movie, or a day-long egocentric recording, the events we care about are usually buried within a sea of irrelevant background noise.

Researchers have shifted toward Temporal Action Localization (TAL) and Video Moment Retrieval (VMR) to solve this "needle-in-a-haystack" problem. The core challenge is not just identifying what is happening, but precisely when it starts and ends amidst hours of "temporal dead zones."

The Signal-to-Noise Challenge

In untrimmed videos, the target action often occupies only a tiny fraction of the total runtime. Hehe Fan and Xiaojun Chang describe this as a "complex event detection" problem where most of a video is filler. For example, in a three-minute video of a "dog show," the defining moments might only last a few seconds.

When models treat every frame as equally important, the "semantic noise" from the background dilutes the signal of the actual event. Qi Zhang and Qin Jin argue that standard global attention mechanisms actually hurt performance in long videos because they force the model to process redundant, non-informative frames. This leads to high false-positive rates, where a model "sees" an action in a mundane background segment simply because it lacks the context to distinguish noteworthy events from routine activity.

Skimming and Previews

To handle the computational cost of processing long videos, researchers have developed "skimming" techniques that mimic how humans browse content. Instead of a "brute force" scan of every frame, these models use lightweight cues to decide where to look closer.

Ruohan Gao, Kristen Grauman, and Lorenzo Torresani introduced a framework called Listen to Look, which uses audio as a "cheap preview." Since audio is much less computationally expensive to process than 3D video features, the model "listens" to the video first to identify promising segments, skipping over the silent or irrelevant parts. Similarly, the Audio-Visual Glance Network (AVGN) by Muhammad Adi Nugroho identifies both when and where (spatial patches) an action occurs to reduce redundant processing.

Walking Through Time

Another intuition for solving sparsity is to treat the video timeline as a path for an agent to explore. Rather than sliding a fixed window across the video, models like SAP (Self-Adaptive Proposal) by Ge Li use reinforcement learning to let an agent "walk" through the video. The agent can dynamically adjust its step size, leaping over "boring" sections and slowing down when it detects a potential action.

This active search strategy is echoed in the work of Wenhao Wu and Dongliang He, who use multiple collaborative agents to sample informative frames. By learning to ignore the "temporal dead zones," these models achieve high accuracy with a fraction of the computational load.

Identifying Signature Moments

In some tasks, the challenge isn't just finding an action, but finding the right action. In skill determination, Hazel Doughty and Dima Damen point out that most frames in a long video of a surgeon or a chef are "neutral." To rank skill levels, a model must learn to attend specifically to "pros" (strengths) and "cons" (mistakes)—sparse moments that define proficiency.

This concept of "signature moves" extends to imitation learning. Hanlin Yang and Jian Yao propose weighting state-action pairs based on Pointwise Mutual Information (PMI). By focusing on the unique moments that define a specific style rather than generic "filler" actions, models can better capture the essence of a behavior from sparse, noisy data.

Denoising the Timeline

The most recent trend involves explicit "denoising" of the video before retrieval. Weijia Liu and colleagues introduced DRNet, which uses the text query to filter out irrelevant clips before the model even attempts to localize the moment. This "denoise-then-retrieve" philosophy acknowledges that in a 10-hour stream—as addressed by the AVA system from Ting Cao and Shiqi Jiang—the primary hurdle is not recognition, but the massive imbalance between relevant signal and background noise.

Papers in this version

Showing 60 of 65 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_localizing_brief_actions_within_long_unt_1ab72237,
  title = {Localizing Brief Actions within Long Untrimmed Videos},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/localizing-brief-actions-within-long-untrimmed-videos/rdr_1ab7223752204158af59cb4ed0031014},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_localizing_brief_actions_within_long_unt_1ab72237_v1,
  title = {Localizing Brief Actions within Long Untrimmed Videos},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/localizing-brief-actions-within-long-untrimmed-videos/rdr_1ab7223752204158af59cb4ed0031014/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}