Abstract
We present a method to learn a joint multimodal representation space that enables recognition of unseen activities in videos. We first compare the effect of placing various constraints on the embedding space using paired text and video data. We also propose a method to improve the joint embedding space using an adversarial formulation, allowing it to benefit from unpaired text and video data. By using unpaired text data, we show the ability to learn a representation that better captures unseen activities. In addition to testing on publicly available datasets, we introduce a new, large-scale text/video dataset. We experimentally confirm that using paired and unpaired data to learn a shared embedding space benefits three difficult tasks: (i) zero-shot activity classification, (ii) unsupervised activity discovery, and (iii) unseen activity captioning, outperforming the state-of-the-art.
Introduction
Standard AI models for activity recognition typically rely on supervised learning, requiring thousands of labeled examples for every specific action. This paradigm fails for the "long tail" of human activities where data is scarce. The challenge lies in Zero-Shot Learning (ZSL): recognizing an activity without having seen a single training video of it.
While text descriptions can bridge this gap—leveraging the fact that we have vast textual knowledge about activities even without corresponding video—most existing methods require perfectly paired video-caption data. Such datasets are expensive and difficult to scale. This research explores whether we can utilize the massive amount of random, unpaired video and text available on the internet to help a model understand activities it has never encountered during training.
Methodology
The core of the proposed approach is a deep autoencoder architecture designed to map both visual and textual inputs into a common latent space ($z$). The system consists of four primary components:
- Video Encoder/Decoder: Processes video frames through a per-segment CNN and temporal attention to map them to the shared embedding.
- Text Encoder/Decoder: Maps sentences into the same latent space using word embeddings and temporal attention.
To ensure that the video and text "speak the same language" in this shared space, the model is trained using a combination of loss functions: Reconstruction Loss (ensuring data can be rebuilt from the embedding), Joint Loss (forcing paired data to be close together), and Cross-domain/Cycle Loss (ensuring consistency when translating between domains).
Adversarial Learning with Unpaired Data
The "secret sauce" of this method is the inclusion of unpaired data—random videos and sentences with no direct relation. To integrate this data, the authors add three "Discriminator" networks. These discriminators act as critics, challenging the encoders to make the latent representations of text and video indistinguishable. This adversarial framework forces the model to learn the underlying semantic structure of activities from any available source, creating a more robust and generalized representation space.
Experimental Setup and Datasets
The model was evaluated on several standard benchmarks and a new dataset introduced by the authors:
- ActivityNet: A large-scale dataset featuring diverse human activities and captions.
- HMDB51 & UCF101: Classic action recognition datasets used to test zero-shot transfer capabilities.
- MLB-YouTube: An expanded version of the baseball dataset, adding approximately 50 hours of aligned announcer commentary to test unsupervised activity discovery.
Key Results and Insights
The researchers found that unpaired data acts as a force multiplier. Incorporating random, unrelated text (such as dictionary definitions) significantly improved the model's ability to navigate the semantic map of unseen videos. Furthermore, the bi-directional nature of the model—mapping both video-to-text and text-to-video—created a more stable and meaningful shared space compared to unidirectional models.
- Zero-Shot Accuracy: On ActivityNet, the full model achieved 82.5% accuracy on unseen classes, significantly outperforming baselines that relied solely on paired data (~76%).
- Cross-Dataset Transfer: When trained on ActivityNet and tested on HMDB51/UCF101, the model outperformed previous state-of-the-art methods by 5-10%.
- Unseen Captioning: The model successfully generated descriptions for activities it never saw during training, with unpaired data improving METEOR scores from 5.31 to 6.89.
Conclusion
By aligning the distributions of visual and textual concepts through adversarial learning, this method effectively bridges the gap between supervised learning and "in-the-wild" activity recognition. While training stability remains a challenge requiring specific "warm-up" algorithms, the results demonstrate that scaling to larger, noisier web-scale data is a promising path forward for truly generalized multimodal AI.
Summary updated