MAC: Mining Activity Concepts for Language-based Temporal Localization

Runzhou Ge, Jiyang Gao, Kan Chen, Ram Nevatia

Abstract

We address the problem of language-based temporal localization in untrimmed videos. Compared to temporal localization with fixed categories, this problem is more challenging as the language-based queries not only have no pre-defined activity list but also may contain complex descriptions. Previous methods address the problem by considering features from video sliding windows and language queries and learning a subspace to encode their correlation, which ignore rich semantic cues about activities in videos and queries. We propose to mine activity concepts from both video and language modalities by applying the actionness score enhanced Activity Concepts based Localizer (ACL). Specifically, the novel ACL encodes the semantic concepts from verb-obj pairs in language queries and leverages activity classifiers' prediction scores to encode visual concepts. Besides, ACL also has the capability to regress sliding windows as localization results. Experiments show that ACL significantly outperforms state-of-the-arts under the widely used metric, with more than 5% increase on both Charades-STA and TACoS datasets.

Introduction

Temporal localization involves identifying the precise start and end times of a specific event within an untrimmed video. While traditional methods rely on a fixed taxonomy of categories (e.g., "running" or "cooking"), language-based temporal localization allows for natural language queries, such as "a person begins opening the refrigerator to find more food." This flexibility introduces significant complexity: there is no predefined list of activities, and queries often contain multiple sub-actions or intricate descriptive details.

Previous state-of-the-art methods typically mapped entire sentence embeddings and global video features into a shared subspace. However, these "black box" approaches often overlook the fine-grained semantic cues—the specific verbs and objects—that define an action. The Mining Activity Concepts (MAC) framework addresses this by explicitly extracting and correlating "activity concepts."

Methodology

The core of the MAC framework is the Activity Concepts based Localizer (ACL), which introduces two primary innovations: the mining of activity concepts and actionness score enhancement.

Mining Activity Concepts

Instead of relying solely on global embeddings, the model extracts specific "concepts" from both modalities to create a more granular representation:

  • Semantic Activity Concepts: The system utilizes a dependency parser to extract verb-object (VO) pairs (e.g., "open refrigerator") from the text query. These pairs are lemmatized and embedded to serve as the core semantic anchor for the search.
  • Visual Activity Concepts: For the video modality, the model leverages the output of a pre-trained action classifier. Rather than using a final hard label, it utilizes the probability distribution across all classes (the fc8 layer) as a feature vector. This captures a "soft" semantic signature of the visual clip.
Figure 1: Conceptual overview of mining activity concepts from language queries and video clips.

The ACL Pipeline Architecture

The architecture employs Multi-modal Processing Units (MPU) to fuse these distinct streams. As illustrated in the pipeline diagram, one MPU processes global features (sentence and video), while another handles the mined activity concepts (VO pairs and action probabilities). The outputs are concatenated to predict the final alignment score and temporal boundary offsets.

Figure 2: The Activity Concepts based Localizer (ACL) architecture.

Actionness Score Enhancement

Sliding window approaches often struggle with "background" clips that contain no meaningful activity. To mitigate this, the authors incorporate an Actionness Score Generator—a binary classifier trained to distinguish between "action" and "background." During inference, the alignment score (the match between video and text) is multiplied by this actionness score, effectively filtering out false positives in the background.

Experiments and Results

The model was evaluated on two primary benchmarks: Charades-STA (everyday indoor activities) and TACoS (cooking-related activities). The ACL model outperformed previous state-of-the-art methods (such as CTRL and ACRN) by a wide margin:

  • Charades-STA: Achieved a 12.20% R@1 (IoU=0.7), compared to 7.15% for the previous best method.
  • TACoS: Achieved a 20.01% R@1 (IoU=0.5), a significant jump over the previous 14.62%.
  • Ablation Studies: Results indicated that using activity concepts alone (without global features) already outperformed previous methods, proving that these mined concepts are highly informative.

Conclusion

The MAC framework demonstrates that explicitly mining semantic and visual concepts significantly improves temporal localization. By focusing on verb-object pairs as semantic anchors and using actionness as a confidence gate, the ACL model reduces localization errors caused by complex queries and busy backgrounds. Future work may involve more sophisticated multi-concept extraction for sentences containing multiple sequential actions.

Summary updated

Abstract

We address the problem of language-based temporal localization in untrimmed videos. Compared to temporal localization with fixed categories, this problem is more challenging as the language-based queries not only have no pre-defined activity list but also may contain complex descriptions. Previous methods address the problem by considering features from video sliding windows and language queries and learning a subspace to encode their correlation, which ignore rich semantic cues about activities in videos and queries. We propose to mine activity concepts from both video and language modalities by applying the actionness score enhanced Activity Concepts based Localizer (ACL). Specifically, the novel ACL encodes the semantic concepts from verb-obj pairs in language queries and leverages activity classifiers' prediction scores to encode visual concepts. Besides, ACL also has the capability to regress sliding windows as localization results. Experiments show that ACL significantly outperforms state-of-the-arts under the widely used metric, with more than 5% increase on both Charades-STA and TACoS datasets.
BibTeX
@inproceedings{ge2019mac,
  author = {Runzhou Ge and Jiyang Gao and Kan Chen and Ram Nevatia},
  title = {{MAC: Mining Activity Concepts for Language-based Temporal Localization}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=p3NqVNC5qO},
}

Loading related content...

Figure