Semi-supervised 3D Video Information Retrieval with Deep Neural Network and Bi-directional Dynamic-time Warping Algorithm

Yintai Ma, Diego Klabjan

Abstract

As digital video content continues to expand exponentially, the ability to efficiently retrieve visually similar content across 2D and 3D domains remains a significant challenge. This paper presents a novel semi-supervised framework designed to address high-dimensional spatial data and complex temporal alignment issues. By integrating deep seq2seq autoencoders with a unique Bi-directional Dynamic Time Warping (Bi-DTW) algorithm, the proposed system achieves state-of-the-art performance in video information retrieval. The framework leverages a two-stage training process—combining unsupervised reconstruction with supervised triplet loss—to optimize feature representation and similarity matching.

Introduction

The proliferation of online video content has turned the task of finding "visually similar" videos into a complex "needle-in-a-haystack" problem. Traditional retrieval methods often falter when faced with two primary obstacles: the high dimensionality of 3D video data (which incorporates depth information) and the "temporal alignment" problem. Temporal alignment is particularly difficult because two videos may depict the identical action but at varying speeds, different frame rates, or even reversed temporal sequences.

While standard Dynamic Time Warping (DTW) is a common solution for sequence matching, its unidirectional nature limits its effectiveness in capturing complex temporal variations. This research introduces a robust system capable of handling large-scale 3D datasets by combining advanced neural architectures with a bi-directional approach to temporal sequence matching.

Methodology

The proposed retrieval pipeline treats video information retrieval as a sequence-matching task, executed through four primary stages:

1. Feature Extraction and Encoding

Videos are partitioned into short clips and processed through a deep neural network encoder. This encoder utilizes LRBP (Long Short-term memory Residual Bidirectional Pooling) blocks. By employing Bidirectional ConvLSTMs, the model is able to capture both spatial and temporal features simultaneously, creating a rich representation of the video content.

2. Seq2Seq Autoencoder Architecture

The system employs a sequence-to-sequence (seq2seq) structure. To accommodate the complexities of 3D video, the researchers introduced specialized components in the decoder:

  • UQB (Quasi 4D CNNs): Designed to handle the additional depth dimension in 3D data.
  • UTB (Transformers): Integrated to manage long-range dependencies and global temporal structures.

3. Semi-supervised Training Strategy

The model undergoes a two-stage training process to maximize its feature-learning capabilities:

  • Unsupervised Pre-training: The model first learns to reconstruct video sequences (autoencoding) on a large-scale dataset. This allows the network to understand general visual and motion features without requiring labeled data.
  • Supervised Fine-tuning: The model is refined using a triplet loss function. This stage explicitly teaches the network to minimize the distance between a "query" video and a "similar" video while maximizing the distance to "dissimilar" samples. A "challenging sample retraining" strategy is used, focusing specifically on the 20% of samples the model finds most difficult to classify.

4. Bi-directional Dynamic Time Warping (Bi-DTW)

The core innovation in the matching process is the Bi-DTW algorithm. Unlike standard DTW, which only calculates similarity in a forward direction, Bi-DTW computes similarity scores for both forward and backward temporal directions. The final similarity score is the minimum of these two values, allowing the system to identify similarities in videos that feature temporal reversals or repetitive, symmetric motions.

Experimental Results

The researchers evaluated the framework (specifically the M3 model variant) across four diverse datasets, including 2D (CC_WEB_VIDEO, YouTube-8m) and 3D (Stanford 3D Indoor Scene, Synthia-SF) data.

Performance Metrics

The M3 model significantly outperformed traditional SIFT-based methods and deep learning benchmarks like NDVR-DML:

  • 2D Retrieval: On the CC_WEB_VIDEO dataset, the model achieved 85.2% mAP (Mean Average Precision), a substantial improvement over the 58.8% achieved by SIFT.
  • Bi-DTW Efficacy: An ablation study highlighted the importance of the matching algorithm; switching from vanilla DTW to Bi-DTW improved accuracy on the CC_WEB_VIDEO dataset from 72.1% to 84.5%.
  • 3D Retrieval: The model proved robust on 3D spatial data, with the M3-3D variant reaching 78.1% mAP on the S3DIS dataset at the clip level.

The Role of Transformers

The inclusion of Transformer units (UTB) in the M3 variant was found to be superior to using ConvLSTMs alone for the decoder. While ConvLSTMs excel at local patterns, Transformers proved essential for capturing the long-range temporal structures necessary for high-accuracy retrieval in complex sequences.

Discussion and Limitations

One of the most significant insights from the study was the effectiveness of Bi-DTW in specific genres. For instance, animation retrieval saw a jump from 92% to 98% mAP, as the bi-directional matching better accounted for the repetitive and symmetric character motions common in the medium.

However, the researchers identified "inter-class confusion" as a remaining challenge. The model occasionally struggled to differentiate between clips with similar visual aesthetics but different semantic meanings—such as comedy and music videos that both feature fast cuts and stage lighting. Furthermore, while Bi-DTW is highly effective, it is more computationally intensive than standard DTW, posing a challenge for real-time applications at a massive scale.

Conclusion

This research demonstrates that combining semi-supervised deep learning with bi-directional temporal matching provides a powerful framework for 3D video retrieval. By addressing both spatial depth and temporal non-linearity, the M3 model sets a new benchmark for accuracy in the field. Future work will likely focus on enhancing semantic understanding to reduce inter-class confusion and optimizing the Bi-DTW algorithm for increased computational efficiency.

Summary updated

BibTeX
@inproceedings{ma2023semi,
  author = {Yintai Ma and Diego Klabjan},
  title = {{Semi-supervised 3D Video Information Retrieval with Deep Neural Network and Bi-directional Dynamic-time Warping Algorithm}},
  booktitle = {IEEE Big Data},
  year = {2023},
  doi = {10.1109/bigdata59044.2023.10386172},
}
Figure