Prompt When the Animal is: Temporal Animal Behavior Grounding with Positional Recovery Training

Sheng Yan, Xin Du, Zongying Li, Yi Wang, Hongcang Jin, Mengyuan Liu

Abstract

Temporal grounding is crucial in multimodal learning, but it poses challenges when applied to animal behavior data due to the sparsity and uniform distribution of moments. To address these challenges, we propose a novel Positional Recovery Training framework (Port), which prompts the model with the start and end times of specific animal behaviors during training. Specifically, Port enhances the baseline model with a Recovering branch to reconstruct corrupted label sequences and align distributions via a Dual-alignment method. This allows the model to focus on specific temporal regions prompted by ground-truth information. Extensive experiments on the Animal Kingdom dataset demonstrate the effectiveness of Port, achieving an [email protected] of 38.52. It emerges as one of the top performers in the sub-track of MMVRAC in ICME 2024 Grand Challenges.

The Challenge of Animal Behavior Grounding

Temporal grounding involves localizing a specific moment in a video based on a natural language query. While significant progress has been made using human-centric datasets, these models often rely on inherent temporal biases. In datasets like Charades-STA or ActivityNet, actions frequently occur at predictable times (e.g., the center of a clip).

In contrast, animal behavior is characterized by its sparsity and uniform distribution. Wildlife footage is often unscripted; an animal might perform a specific action at any point in a long video, and these actions are typically very short. Analysis shows that the normalized moment length in the Animal Kingdom dataset is only 0.19, significantly lower than the 0.32 seen in ActivityNet. This creates a "needle in a haystack" scenario where traditional models, lacking strong temporal priors, struggle to perform.

Temporal distribution comparison between human-centric datasets and Animal Kingdom.

The Port Framework

To overcome the lack of temporal bias, we introduce Port, a framework utilizing Positional Recovery Training (PRT). Port builds upon the VSLNet baseline by adding a secondary "Recovering" branch that acts as a teacher during the training phase.

Architecture and Dual-Alignment

The Port architecture consists of two parallel branches:

  1. Predicting Branch: This standard branch attempts to regress the start and end times of an action using combined video and text features.
  2. Recovering Branch: During training, this branch is "prompted" with a corrupted version of the ground-truth labels (where start/end tokens are randomly flipped). Its objective is to reconstruct the original, accurate labels.
  3. Dual-alignment: The Recovering branch, having access to noisy ground truth, generates sharper probability distributions. Port employs Kullback-Leibler (KL) divergence to align the Predicting branch's output with these superior distributions.
Architecture of the Positional Recovery Training (Port) framework.

By using the Recovering branch as a positional prompt, the model learns to focus on correct temporal regions. Crucially, the Recovering branch is disabled during inference, ensuring the final model remains efficient and does not require ground-truth assistance for real-world predictions.

Experimental Results

The Port framework was evaluated on the Animal Kingdom dataset, which features 50 hours of diverse wildlife behavior across 18,000+ annotations. Port achieved state-of-the-art results, significantly outperforming existing benchmarks:

Ablation studies confirmed that the Dual-alignment mechanism is the primary driver of these improvements. Without the link between the two branches, performance regressed toward the baseline, proving that the positional "prompting" effectively teaches the model to navigate unbiased temporal data.

Conclusion and Future Work

Port demonstrates that the "prompting" paradigm can successfully mitigate the challenges of sparse and uniformly distributed data in animal behavior grounding. While highly effective, future work will address limitations in handling extremely long videos—where small indexing errors lead to large timestamp instabilities—and integrate classification branches to better distinguish between similar behaviors in complex wildlife scenes.

Summary updated

Abstract

Temporal grounding is crucial in multimodal learning, but it poses challenges when applied to animal behavior data due to the sparsity and uniform distribution of moments. To address these challenges, we propose a novel Positional Recovery Training framework (Port), which prompts the model with the start and end times of specific animal behaviors during training. Specifically, \port{} enhances the baseline model with a Recovering branch to reconstruct corrupted label sequences and align distributions via a Dual-alignment method. This allows the model to focus on specific temporal regions prompted by ground-truth information. Extensive experiments on the Animal Kingdom dataset demonstrate the effectiveness of \port{}, achieving an [email protected] of 38.52. It emerges as one of the top performers in the sub-track of MMVRAC in ICME 2024 Grand Challenges.
BibTeX
@misc{yan2024prompt,
  author = {Sheng Yan and Xin Du and Zongying Li and Yi Wang and Hongcang Jin and Mengyuan Liu},
  title = {{Prompt When the Animal is: Temporal Animal Behavior Grounding with Positional Recovery Training}},
  year = {2024},
  eprint = {2405.05523},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.05523},
}
Figure