Abstract
Temporal action localization in instructional videos is hindered by noisy narrations and the lack of large-scale manual annotations. This paper introduces a novel framework that utilizes Large Language Models (LLMs) to "denoise" ASR transcripts into task-relevant "LLM-steps." To align these steps with video segments, we propose a Multi-Pathway Text-Video Alignment (MPTVA) strategy that fuses temporal information from narration timestamps, long-term semantic similarity, and short-term fine-grained visual features. Experimental results demonstrate that our method achieves state-of-the-art performance across multiple downstream tasks, including procedure step grounding and action localization.
Introduction: The Challenge of Noisy Narrations
Instructional videos are a rich source of procedural knowledge, but training models to accurately localize specific steps (e.g., "dice the avocado") is difficult. Standard methods rely on Automatic Speech Recognition (ASR) transcripts, which suffer from two primary issues:
- Irrelevant Content: Narrators often include conversational "chatter" (e.g., "Kale is my favorite vegetable") that does not describe a physical action.
- Temporal Misalignment: Spoken instructions often precede or follow the actual visual action, leading to unreliable timestamps.
While external knowledge bases like wikiHow can provide clean procedural steps, they often lack the video-specific nuances required for precise alignment.
Methodology
The proposed framework addresses these issues through a two-stage process: LLM-based step extraction and Multi-Pathway Text-Video Alignment (MPTVA).
1. LLM-Based Step Extraction
To overcome the limitations of raw transcripts and generic guides, the framework employs an LLM (such as Llama 2) to process ASR narrations. The LLM is prompted to filter out colloquialisms and summarize the core procedural actions into a list of LLM-steps. This process ensures the text is both clean and specific to the actions performed in the video.
2. Multi-Pathway Text-Video Alignment (MPTVA)
To generate reliable training labels without manual intervention, the system fuses three distinct alignment pathways to create "pseudo-labels":
- S-N-V (Step-Narration-Video): Aligns LLM-steps to narrations and leverages existing ASR timestamps.
- S-V-Long: Uses a model pre-trained on long-form instructional data to find global semantic matches between steps and video segments.
- S-V-Short: Employs a video foundation model (InternVideo) to capture fine-grained, short-term interactions between humans and objects.
By identifying points of "mutual agreement" among these pathways, the framework effectively filters out errors unique to any single method, resulting in a robust training signal for the final Transformer-based model.
Evaluation and Results
The framework was trained on the HTM-370K dataset and evaluated on benchmarks including HT-Step, CrossTask, and HTM-Align.
The proposed method established new state-of-the-art (SOTA) results across three major tasks:
- Procedure Step Grounding: Improved by 5.9% over previous SOTA.
- Action Step Localization: Improved by 3.1%.
- Narration Grounding: Improved by 2.8%.
Ablation studies confirmed that the multi-pathway approach is critical, providing a 4-5% performance boost compared to using any single alignment pathway.
Key Insights
- LLMs as Contextual Denoising Tools: LLMs are superior to static guides like wikiHow because they can summarize the specific actions occurring in a particular video context.
- Robustness through Fusion: The MPTVA strategy effectively mitigates the noise inherent in ASR timestamps and semantic matching by requiring cross-pathway consensus.
- Cross-Domain Knowledge: Integrating general-purpose video foundation models (S-V-Short) provides the fine-grained visual detail often missing in models trained solely on instructional data.
Limitations and Future Work
The system's efficacy depends on hyperparameter tuning for filtering thresholds and window sizes. Furthermore, the pre-processing stage involving LLMs and multiple foundation models is computationally intensive. Future research may focus on optimizing these costs or developing more integrated LLM-visual reasoning architectures.
Summary updated