Energy vs. Noise: Towards Robust Temporal Action Localization in Open-World

Chenyu Mu, Jiahua Li, Kun Wei, Cheng DengAAAI 2025 · OpenReview

Introduction

Temporal Action Localization (TAL) is the task of identifying the precise start and end times of actions in untrimmed videos and assigning them a category. While state-of-the-art models perform well on carefully curated datasets, they are notoriously fragile in "open-world" scenarios where human annotations are often inconsistent. This inconsistency introduces two primary types of label noise: Boundary Noise, where annotators disagree on exact start/end points, and Category Noise, where similar actions (e.g., a baseball pitch vs. a cricket bowl) are confused.

The subjectivity of temporal action boundary annotations and the resulting performance drop in existing models under various noise conditions.

As illustrated in the figure above, even a small amount of mixed noise causes a significant performance drop in leading models like ActionFormer and TriDet. Current TAL methods lack a mechanism to distinguish between true action patterns and the noise introduced by imperfect human labeling. To address this, the authors introduce the Energy-Driven Meta Purifier (EDMP), a plug-and-play framework that uses energy-based modeling and meta-learning to automatically correct label errors and reweight training samples.

The EDMP Framework

EDMP operates by treating label correction as a "meta-task." It uses a tiny, meticulously annotated "meta-set" (as little as 2% of the total data) to teach the model how to fix noisy labels in the larger training set. Borrowing from physics-inspired energy-based learning, the system defines a function that maps video features to "energy" values, allowing it to distinguish between actual actions and surrounding context without relying solely on potentially incorrect labels.

Overview of the Energy-Driven Meta Purifier (EDMP) framework, featuring the Semantic Purification (SP), Temporal Refinement (TR), and Energy Synergy Optimizer (ESO) modules.

The architecture is composed of three main pillars:

  • Semantic Purification Module (SPM): Corrects category mislabeling by reweighting samples based on their energy signatures.
  • Temporal Refinement Module (TRM): Adjusts start and end times using a "Boundary Energy Loss" to ensure refined boundaries represent a sharp transition in energy between the action and the background.
  • Energy Synergy Optimizer (ESO): A global reweighting mechanism that exploits the "memorization effect" (where models learn clean patterns before overfitting to noise) to assign lower weights to highly suspicious samples during training.

Key Results and Insights

The researchers evaluated EDMP on industry-standard benchmarks, THUMOS14 and ActivityNet 1.3, by systematically injecting synthetic noise.

  • Robustness Gains: EDMP consistently outperformed baseline models across all noise levels. On THUMOS14 with heavy mixed noise, EDMP improved the average mAP (mean Average Precision) of ActionFormer by 5.79% and TemporalMaxer by 6.43%.
  • State-of-the-Art Comparison: When compared to other "Learning with Noisy Labels" (LNL) methods like Co-teaching, EDMP achieved the highest performance (58.35% mAP vs. 54.87%).
  • Efficiency: The authors demonstrate that a very small clean set (e.g., 20 videos for THUMOS14) is sufficient to guide the purification process effectively.

Limitations and Future Work

While EDMP is a powerful tool for robust TAL, it does assume the existence of a small, perfectly clean meta-set, which might be difficult to acquire in some niche domains. Additionally, the meta-learning approach involves bi-level optimization, which increases training time and memory requirements. Future work may explore applying this energy-based approach to Weakly Supervised TAL, where no boundary information is provided during training.

Summary updated

BibTeX
@inproceedings{mu2025energy,
  author = {Chenyu Mu and Jiahua Li and Kun Wei and Cheng Deng},
  title = {{Energy vs. Noise: Towards Robust Temporal Action Localization in Open-World}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=Y0eHUuxlql},
}
Figure