Detection of Important Scenes in Baseball Videos via Bidirectional Time Lag Aware Deep Multiset Canonical Correlation Analysis

Kaito Hirasawa, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama

Introduction

Baseball broadcasts often exceed three hours, creating a significant demand for automated highlight generation. Traditional AI approaches to this task face two primary challenges: the "labeling burden" of manually annotating thousands of hours of footage and the "Twitter lag"—the temporal disconnect between an on-field event and the subsequent surge in social media reactions. Fans do not tweet instantaneously; there is a delay for processing and typing, and a single event can trigger a prolonged flurry of discussion.

To address these issues, researchers have developed BiTl-dMCCA (Bidirectional Time Lag Aware Deep Multiset Canonical Correlation Analysis), an unsupervised framework that identifies highlights as statistical anomalies by syncing game footage with real-time Twitter data.

The BiTl-dMCCA Framework

The proposed architecture operates in four distinct stages, moving from raw data to the identification of important scenes.

Overview of the proposed BiTl-dMCCA framework for important scene detection.
  1. Multi-modal Feature Extraction: The system processes three synchronized streams: textual data (tweets via Tweet2Vec), visual data (video shots via 3D ResNet), and audio data (spectrograms via VGG16).
  2. Bidirectional Correlation Maximization: This core component aligns the disparate data types into a shared mathematical embedding space while accounting for temporal offsets.
  3. GAN-based Model Construction: A Generative Adversarial Network (MAD-GAN) is trained to recognize "normal" gameplay patterns.
  4. Important Scene Detection: During testing, scenes that the GAN cannot accurately reconstruct—indicating a deviation from the norm—are flagged as highlights.

Modeling the "Twitter Lag"

A key innovation of this research is the bidirectional modeling of time lags. Standard correlation methods often assume data points are perfectly synchronized. However, in sports, an event influences future tweets, and conversely, a tweet at a specific time is influenced by past events.

Illustration of the bidirectional time-lag influence between on-field events and social media reactions.

The authors utilize a Poisson distribution to weight these influences over time. This acknowledges that the peak of social media chatter typically occurs shortly after a play and then gradually tapers off. By considering both directions of influence, the model more accurately captures the relationship between the game's visual/audio cues and the audience's textual reactions.

Unsupervised Anomaly Detection

To eliminate the need for manual labeling, the researchers reframed highlight detection as an anomaly detection problem. They constructed a "normal" training dataset by calculating the median of features across 23 different baseball matches. Because highlights (like home runs or spectacular catches) are statistically rare, the median effectively represents the standard, "boring" baseline of a broadcast.

The GAN is trained on this median data. When the model encounters a highlight during a new game, its reconstruction error spikes because the highlight's features differ significantly from the learned "normal" baseline. These high-error moments are then identified as the most important scenes.

Results and Insights

The BiTl-dMCCA method was tested against 15 baseline models using a dataset of 28 baseball matches. Key findings include:

  • Superior Performance: The proposed method achieved a specificity of 0.402 at maximum sensitivity, outperforming unimodal models and those that ignored time lags.
  • Bidirectional Advantage: Results confirmed that looking "both ways" (how events affect future tweets and how tweets relate to past events) provides a more robust alignment than unidirectional models.
  • Highlights as Deviations: The study proves that defining a "normal game" is a highly effective way to discover exciting moments without requiring subjective human labels.

Conclusion

While the current model relies on empirically set time-lag parameters, it represents a significant step toward scalable, automated sports broadcasting tools. Future work aims to automatically adjust these parameters based on specific event types and optimize feature extraction for real-time applications.

Summary updated

Abstract

A novel method for detection of important scenes in baseball videos based on correlation maximization between heterogeneous modalities via bidirectional time lag aware deep multiset canonical correlation analysis (BiTl-dMCCA) is presented in this paper. The proposed method enables detection of important scenes by collaboratively using baseball videos and their corresponding tweets. The technical contributions of this paper are twofold. First, since there are time lags between not only “tweets and corresponding multiple previous events” but also “events and corresponding multiple following posted tweets”, the proposed method considers these bidirectional time lags. Specifically, the representation of such bidirectional time lags into the derivation of their covariance matrices is newly introduced. Second, the proposed method adopts textual, visual and audio features calculated from tweets and videos as multi-modal time series features. Important scenes are detected as abnormal scenes via anomaly detection based on a generative adversarial network using multi-modal features projected by BiTl-dMCCA. The proposed method does not need any training data with annotation. Experimental results obtained by applying the proposed method to actual baseball matches show the effectiveness of the proposed method.
BibTeX
@article{hirasawa2021detection,
  author = {Kaito Hirasawa and Keisuke Maeda and Takahiro Ogawa and Miki Haseyama},
  title = {{Detection of Important Scenes in Baseball Videos via Bidirectional Time Lag Aware Deep Multiset Canonical Correlation Analysis}},
  journal = {IEEE Access},
  year = {2021},
  doi = {10.1109/access.2021.3088284},
}
Figure