Exploring Temporal Concurrency for Video-Language Representation Learning

Heng Zhang, Daqing Liu, Zezhong Lv, Bing Su, Dacheng Tao

Introduction: Beyond Discrete Bags of Clips

Traditional video-language models often treat video-paragraph pairs as discrete collections—matching a "bag of video clips" against a "bag of sentences." This approach frequently disrupts the internal temporal logic of each modality and fails to capture the global alignment between visual and textual streams.

The authors of this paper propose a shift in perspective: viewing video and text as Temporal Concurrent Processes (TCP). In this view, both modalities are parallel, continuous sequences that evolve smoothly over time. This paradigm shift allows the model to capture long-range dependencies and maintain the natural "flow" of information across both visual and linguistic streams.

Conceptual evolution from simple semantic alignment to temporal concurrency.

The TCP Framework: Alignment and Regularization

The TCP framework introduces two core innovations to model these concurrent processes effectively: cross-modal sequence alignment and intra-modal sequence modeling.

1. Cross-modal Sequence Alignment via soft-DTW

To find the optimal alignment between a video and its corresponding paragraph—where actions and descriptions may proceed at different paces—the model employs soft Dynamic Time Warping (soft-DTW). Unlike standard DTW, which is non-differentiable, soft-DTW provides a smooth, differentiable cost function. This allows the neural network to learn the optimal global alignment between the two sequences during end-to-end training, effectively mapping the "minimum cost path" between sentence and video segment embeddings.

2. Intra-modal Sequence Modeling with the Brownian Bridge

A common risk in global alignment is "representation collapse," where the model maps all embeddings to a single point to minimize distance. To prevent this, TCP utilizes a Process-wised Regularization Term (PRT) based on the Brownian bridge stochastic process. This constraint ensures that any intermediate representation remains a predictable "bridge" between the start and end points of a sequence. By enforcing this temporal continuity, the model preserves the inherent dynamics of the video and the narrative flow of the text, ensuring that the latent space reflects the actual evolution of the content.

The TCP architecture, highlighting the dual-encoder setup, soft-DTW cost matrix, and Brownian bridge regularization.

Experimental Results and Benchmarks

The model was pre-trained on the LF-VILA-8M dataset, which contains 8.5 million long-form video-paragraph pairs. Unlike datasets like HowTo100M, which can be noisy, LF-VILA-8M is noted for its cleaner annotations and longer average durations, making it ideal for testing long-range temporal modeling. Evaluation across major benchmarks demonstrated that TCP sets new state-of-the-art (SOTA) standards:

  • Video Retrieval: On MSR-VTT (1K test set), TCP achieved a Recall@1 of 38.0 in fine-tuning and 26.8 in zero-shot settings, outperforming previous leaders like MILES and HD-VILA.
  • Video Question Answering: The model reached 92.8% accuracy on MSR-VTT and 83.7% on LSMDC, surpassing existing "All-in-one" models.
  • Scalability: Performance scales effectively with data, showing that the architecture benefits significantly from larger pre-training corpora without hitting an immediate performance ceiling.

Conclusion and Future Work

By treating video and language as concurrent temporal processes rather than static samples, the TCP framework captures long-range dependencies and maintains temporal coherence more effectively than previous methods. While future work may explore more complex stochastic processes or finer-grained (pixel-level) tracking, TCP proves that modeling the process of data evolution is a powerful paradigm for multi-modal learning. The efficiency of the model—achieving superior results with fewer pairs than massive 100M+ datasets—underscores the importance of structural modeling over raw scale.

Summary updated

Abstract

Paired video and language data is naturally temporal concurrency, which requires the modeling of the temporal dynamics within each modality and the temporal alignment across modalities simultaneously. However, most existing video-language representation learning methods only focus on discrete semantic alignment that encourages aligned semantics to be close in the latent space, or temporal context dependency that captures short-range coherence, failing in building the temporal concurrency. In this paper, we propose to learn video-language representations by modeling video-language pairs as Temporal Concurrent Processes (TCP) via a process-wised distance metric learning framework. Specifically, we employ the soft Dynamic Time Warping (DTW) to measure the distance between two processes across modalities and then optimize the DTW costs. Meanwhile, we further introduce a regularization term that enforces the embeddings of each modality approximating a stochastic process to guarantee the inherent dynamics. Experimental results on three benchmarks demonstrate that TCP stands as a state-of-the-art method for various video-language understanding tasks, including paragraph-to-video retrieval, video moment retrieval, and video question-answering. Code is available at https: //github.com/hengRUC/TCP.
BibTeX
@inproceedings{zhang2023exploring,
  author = {Heng Zhang and Daqing Liu and Zezhong Lv and Bing Su and Dacheng Tao},
  title = {{Exploring Temporal Concurrency for Video-Language Representation Learning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01427},
}
Figure