Time-Lag Aware Latent Variable Model for Prediction of Important Scenes Using Baseball Videos and Tweets

Kaito Hirasawa, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama

TL;DR

This paper introduces Tl-LVM, a novel end-to-end model that predicts important scenes in baseball games by jointly analyzing video and associated tweets. Its key innovation is a time-lag aware loss function that accounts for the delay between game events and viewer reactions on Twitter, leading to more accurate predictions.

What Problem Does This Paper Address?

Watching sports, especially long games like baseball, can be time-consuming. Generating highlights of key moments (like home runs or RBIs) is a common solution, but manual selection is slow. Predicting these important scenes in real-time allows viewers to efficiently follow the game. Previous methods often relied solely on video analysis or struggled to incorporate the rich, real-time commentary from social media like Twitter. A significant challenge is the inherent time lag: tweets are often posted after an event occurs, and a single tweet might reflect multiple preceding events. This paper aims to bridge this gap by developing a model that effectively leverages both video and tweet data while explicitly accounting for these time lags.

Methods

The core of the proposed method is a Time-Lag Aware Latent Variable Model (Tl-LVM), built upon a Multimodal Variational Autoencoder (MVAE). Here's a breakdown:

  • Multimodal Input: Tl-LVM takes both baseball video (visual and audio features) and associated Twitter data as input.
  • Encoder-Decoder Architecture: Similar to standard VAEs, it uses an encoder to extract latent features and a decoder to reconstruct the original inputs.
  • Important Scene Predictor: A dedicated component predicts the probability of a given scene being "important."
  • Novel Time-Lag Loss Function: This is the paper's main contribution. Instead of treating tweets and video events as simultaneous, Tl-LVM introduces a loss term that explicitly models the correlation between tweet features and past video features, weighted by a Poisson distribution to represent the decaying influence of older events. This allows the model to learn relationships across time.
  • End-to-End Training: Crucially, the encoder, decoder, and important scene predictor are trained simultaneously using this novel loss function. This contrasts with prior work that might train these components separately or only partially account for time lags.

Datasets

The authors used a private dataset consisting of:

  • 12 baseball videos from Pacific League TV (broadcast between June and September 2019).
  • Associated tweets collected using the hashtag "#lovefighters".

Seven games were used for training, and five for testing. This dataset was chosen because, according to the authors, public datasets are not readily available for this specific task, and their novelty cannot be verified with existing benchmarks.

Key Insights and Ideas

  • Tweets as a Rich Source of Context: The paper strongly supports the idea that social media reactions (tweets) provide valuable, real-time context that significantly enhances the prediction of important sports events, complementing visual information.
  • Time Lags are Critical: Ignoring the temporal gap between events and reactions leads to suboptimal performance. Explicitly modeling these time lags is key to accurate prediction.
  • End-to-End Learning for Synergy: Simultaneously training all components (encoder, decoder, predictor) allows for better feature extraction and prediction, as the model can optimize for all objectives at once.
  • Poisson Distribution for Influence Decay: Using a Poisson distribution to model the influence of past events on current tweets is an intuitive and effective way to capture how recent events have a stronger impact than older ones.

Key Results

  • Superior Performance: Tl-LVM consistently outperformed several baseline methods (Comps. 1-11) in terms of F-measure, a common metric for evaluating prediction tasks. For instance, Tl-LVM achieved an average F-measure of 0.441, compared to the next best (Comp. 11) at 0.436.
  • Effectiveness of Multimodality: The results showed that using tweets, visual, and audio features together (as in Tl-LVM) was more effective than using individual modalities or simpler combinations.
  • Time-Lag Awareness Matters: Tl-LVM's F-measure was higher than Comp. 8, which used MVAE but did not consider time lags, highlighting the importance of this feature.
  • End-to-End Advantage: Tl-LVM also outperformed Comp. 11, which separately trained feature transformation and prediction networks, demonstrating the benefit of end-to-end training.
  • Improved Specificity: When evaluated for specificity at near-perfect sensitivity (meaning it accurately identified almost all important scenes while minimizing false positives), Tl-LVM showed a 1.6% improvement over Comp. 11, indicating better suppression of normal scene predictions.

Limitations and Future Work

  • Private Dataset: The use of a private dataset limits the reproducibility and generalizability of the findings.
  • Parameter Tuning: While parameters were tuned for optimal f-measure, the paper suggests that future work could focus on automatically determining parameters of the Poisson distribution (λ and L) based on event characteristics.
  • Automatic Parameter Determination: The authors propose developing a loss function that can flexibly consider these automatically determined parameters for even better performance.

Summary updated

Abstract

In this study, a novel prediction method for predicting important scenes in baseball videos using a time-lag aware latent variable model (Tl-LVM) is proposed. Tl-LVM adopts a multimodal variational autoencoder using tweets and videos as the latent variable model. It calculates the latent features from these tweets and videos and predicts important scenes using these latent features. Since time lags exist between posted tweets and events, Tl-LVM introduces the loss considering time lags by correlating the feature into the loss function of the multimodal variational autoencoder. Furthermore, Tl-LVM can train the encoder, decoder, and important scene predictor, simultaneously, using this loss function. This is the novelty of Tl-LVM, and this work is the first end-to-end prediction model of important scenes that considers time lags to the best of our knowledge. It is the contribution of Tl-LVM to realize high-quality prediction using latent features that consider time lags between tweets and multiple corresponding previous events. Experimental results using actual tweets and baseball videos show the effectiveness of Tl-LVM.
BibTeX
@inproceedings{hirasawa2022time,
  author = {Kaito Hirasawa and Keisuke Maeda and Takahiro Ogawa and Miki Haseyama},
  title = {{Time-Lag Aware Latent Variable Model for Prediction of Important Scenes Using Baseball Videos and Tweets}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=UX4egHRx4k},
}
Figure