Learning to Localize Actions in Instructional Videos with LLM-Based Multi-Pathway Text-Video Alignment

Yuxiao Chen, Kai Li, Wentao Bao, Deep Patel, Yu Kong, Martin Renqiang Min, Dimitris N. Metaxas

Abstract

Temporal action localization in instructional videos is hindered by noisy narrations and the lack of large-scale manual annotations. This paper introduces a novel framework that utilizes Large Language Models (LLMs) to "denoise" ASR transcripts into task-relevant "LLM-steps." To align these steps with video segments, we propose a Multi-Pathway Text-Video Alignment (MPTVA) strategy that fuses temporal information from narration timestamps, long-term semantic similarity, and short-term fine-grained visual features. Experimental results demonstrate that our method achieves state-of-the-art performance across multiple downstream tasks, including procedure step grounding and action localization.

Introduction: The Challenge of Noisy Narrations

Instructional videos are a rich source of procedural knowledge, but training models to accurately localize specific steps (e.g., "dice the avocado") is difficult. Standard methods rely on Automatic Speech Recognition (ASR) transcripts, which suffer from two primary issues:

  1. Irrelevant Content: Narrators often include conversational "chatter" (e.g., "Kale is my favorite vegetable") that does not describe a physical action.
  2. Temporal Misalignment: Spoken instructions often precede or follow the actual visual action, leading to unreliable timestamps.

While external knowledge bases like wikiHow can provide clean procedural steps, they often lack the video-specific nuances required for precise alignment.

Methodology

The proposed framework addresses these issues through a two-stage process: LLM-based step extraction and Multi-Pathway Text-Video Alignment (MPTVA).

Overview of the proposed framework, illustrating (a) LLM-step extraction from narrations, (b) the MPTVA strategy for generating pseudo-matches, and (c) the final Transformer-based model architecture.

1. LLM-Based Step Extraction

To overcome the limitations of raw transcripts and generic guides, the framework employs an LLM (such as Llama 2) to process ASR narrations. The LLM is prompted to filter out colloquialisms and summarize the core procedural actions into a list of LLM-steps. This process ensures the text is both clean and specific to the actions performed in the video.

Comparison of raw narrations, generic wikiHow steps, and the proposed LLM-steps, highlighting how LLMs effectively filter noise and improve relevance.

2. Multi-Pathway Text-Video Alignment (MPTVA)

To generate reliable training labels without manual intervention, the system fuses three distinct alignment pathways to create "pseudo-labels":

  • S-N-V (Step-Narration-Video): Aligns LLM-steps to narrations and leverages existing ASR timestamps.
  • S-V-Long: Uses a model pre-trained on long-form instructional data to find global semantic matches between steps and video segments.
  • S-V-Short: Employs a video foundation model (InternVideo) to capture fine-grained, short-term interactions between humans and objects.

By identifying points of "mutual agreement" among these pathways, the framework effectively filters out errors unique to any single method, resulting in a robust training signal for the final Transformer-based model.

Evaluation and Results

The framework was trained on the HTM-370K dataset and evaluated on benchmarks including HT-Step, CrossTask, and HTM-Align.

The proposed method established new state-of-the-art (SOTA) results across three major tasks:

  • Procedure Step Grounding: Improved by 5.9% over previous SOTA.
  • Action Step Localization: Improved by 3.1%.
  • Narration Grounding: Improved by 2.8%.

Ablation studies confirmed that the multi-pathway approach is critical, providing a 4-5% performance boost compared to using any single alignment pathway.

Key Insights

  • LLMs as Contextual Denoising Tools: LLMs are superior to static guides like wikiHow because they can summarize the specific actions occurring in a particular video context.
  • Robustness through Fusion: The MPTVA strategy effectively mitigates the noise inherent in ASR timestamps and semantic matching by requiring cross-pathway consensus.
  • Cross-Domain Knowledge: Integrating general-purpose video foundation models (S-V-Short) provides the fine-grained visual detail often missing in models trained solely on instructional data.

Limitations and Future Work

The system's efficacy depends on hyperparameter tuning for filtering thresholds and window sizes. Furthermore, the pre-processing stage involving LLMs and multiple foundation models is computationally intensive. Future research may focus on optimizing these costs or developing more integrated LLM-visual reasoning architectures.

Summary updated

Abstract

Learning to localize temporal boundaries of procedure steps in instructional videos is challenging due to the limited availability of annotated large-scale training videos. Recent works focus on learning the cross-modal alignment between video segments and ASR-transcripted narration texts through contrastive learning. However, these methods fail to account for the alignment noise, i.e., irrelevant narrations to the instructional task in videos and unreliable timestamps in narrations. To address these challenges, this work proposes a novel training framework. Motivated by the strong capabilities of Large Language Models (LLMs) in procedure understanding and text summarization, we first apply an LLM to filter out task-irrelevant information and summarize task-related procedure steps (LLM-steps) from narrations. To further generate reliable pseudo-matching between the LLM-steps and the video for training, we propose the Multi-Pathway Text-Video Alignment (MPTVA) strategy. The key idea is to measure alignment between LLM-steps and videos via multiple pathways, including: (1) step-narration-video alignment using narration timestamps, (2) direct step-to-video alignment based on their long-term semantic similarity, and (3) direct step-to-video alignment focusing on short-term fine-grained semantic similarity learned from general video domains. The results from different pathways are fused to generate reliable pseudo step-video matching. We conducted extensive experiments across various tasks and problem settings to evaluate our proposed method. Our approach surpasses state-of-the-art methods in three downstream tasks: procedure step grounding, step localization, and narration grounding by 5.9\%, 3.1\%, and 2.8\%.
BibTeX
@inproceedings{chen2024learning,
  author = {Yuxiao Chen and Kai Li and Wentao Bao and Deep Patel and Yu Kong and Martin Renqiang Min and Dimitris N. Metaxas},
  title = {{Learning to Localize Actions in Instructional Videos with LLM-Based Multi-Pathway Text-Video Alignment}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-73007-8\_12},
}
Figure