Research on Analysis of Sports Video Multi-Pattern Fusion

Jia Wang, Haitao Yang, Yang Wang, Jingmeng SunInternational Journal of Database Theory and Application 2015 · DOI

TL;DR

This paper introduces a unified probabilistic framework for sports video analysis that effectively integrates multimodal information (like visual and audio cues) and hierarchical event relationships. Their proposed Product Hierarchical Hidden Markov Model (PHHMM) outperforms existing methods in detecting and analyzing semantic events in sports videos.

What Problem Does This Paper Address?

Analyzing sports videos to understand semantic events (like "goal," "foul," or "in-play" vs. "out-of-play") is challenging because these events are inherently multimodal. They are conveyed through various cues: visual information (player actions, camera angles, on-field graphics), audio (commentary, crowd noise), and textual information (subtitles, scores). Previous methods often focused on single modalities or treated events in isolation. However, sports events are sequential and have contextual relationships, meaning understanding one event often depends on what happened before and after. This paper aims to address this by developing a framework that can fuse information from multiple modalities and model the hierarchical and temporal relationships between events for more effective analysis.

Methods

The core of this work is a unified probabilistic framework based on Dynamic Bayesian Networks (DBNs). DBNs are powerful for modeling sequential data and probabilistic relationships. Building on this, the authors propose three specific multi-level, multi-modal models:

  • Factorial Hierarchical Hidden Markov Model (FHHMM)
  • Coupled Hierarchical Hidden Markov Model (CHHMM)
  • Product Hierarchical Hidden Markov Model (PHHMM)

These models extend traditional Hierarchical Hidden Markov Models (HHMMs) by allowing for the integration of multiple observation patterns (modalities) at different hierarchical levels. The PHHMM, in particular, is designed to effectively combine these modalities. For inference (event detection), they transform the DBN models into a standard HMM and apply the Viterbi algorithm, which is a common and efficient method for finding the most likely sequence of hidden states.

Datasets

The paper uses a dataset of 20 video clips of football matches. These clips are in MPEG-1 format, with a resolution of 352x288 at 25 frames per second, and range from a few minutes to over ten minutes in length. The data was manually annotated for "in-play" and "out-of-play" events. The authors mention that features were extracted every 0.5 seconds to reduce computational load.

Key Insights and Ideas

  • Multimodality is Crucial: Analyzing sports events effectively requires fusing information from various sources (visual, audio, text), as no single modality tells the whole story.
  • Hierarchy Matters: Sports events have a natural hierarchical structure (e.g., a "goal" event might be composed of sub-events like "ball kicked," "ball crosses line"). Modeling this hierarchy is key to deeper understanding.
  • Context is King: Events are not isolated. Their temporal and consequential relationships are vital for accurate detection and recognition.
  • DBNs as a Unified Framework: Dynamic Bayesian Networks provide a flexible and powerful probabilistic foundation for integrating multimodal information and hierarchical structures.
  • PHHMM's Superiority: The Product Hierarchical Hidden Markov Model (PHHMM) demonstrated the best performance, suggesting that its specific way of combining modalities and hierarchical levels is particularly effective for sports video analysis.

Key Results

The paper presents experimental results based on two evaluation metrics: frame-based accuracy and fragment-based evaluation (using Precision, Recall, and F-value).

  • Frame-Based Accuracy: The PHHMM achieved the highest accuracy rate of 84.6%. This was significantly better than single-mode HHMMs (e.g., HHMM_C at 78.6%, HHMM_M at 63.7%) and also outperformed other proposed multi-modal models like FHHMM (81.4%) and CHHMM (86.3%).
  • Fragment-Based Evaluation (F-value): The PHHMM achieved the highest F-value of 80.6%. This metric combines precision and recall. While FHHMM and CHHMM showed high recall (100% and 99.3% respectively), their precision was lower (36.4% and 36.3%), indicating over-segmentation. The PHHMM balanced these well.
  • Overall Improvement: Compared to single-mode HHMM, the proposed multi-level models (especially PHHMM) showed performance increases of about 15% in both frame-based and fragment-based evaluations.

Limitations and Future Work

  • Limitations:
    • The paper primarily focuses on visual and motion features, with audio features not being used due to perceived lack of distinguishability for "in-play" vs. "out-of-play" detection in their observation.
  • The performance of their models can be sensitive to the number of states chosen for the hidden variables; finding the optimal number requires careful tuning.
  • The feature fusion method used in some benchmark models was found to be worse than single-mode analysis, possibly due to interference from motion features.
  • Future Work:
    • The authors suggest that PHHMM is a promising semantic analysis model, implying further exploration and application of this model.
  • While not explicitly stated as future work, the success of PHHMM suggests that further research into more sophisticated fusion techniques and the integration of other modalities (like audio and text) could yield even better results.
  • The paper mentions that the observation probability can be represented by other forms besides multivariate Gaussian models, hinting at potential extensions.

Summary updated

Abstract

In order to effectively integrate multimodal information and multilayer constraints, we present a unified probabilistic framework for sports video analysis. Based the framework, three instances of statistical models are constructed and compared. Experimental results indicate our method with multimodal fusion processes semantic events in sports video more effectively. With the method based on statistics, we'll discuss further sport video content analysis method fusing multimode information. Semantic events in sport videos are in essence multimodal. In television relay, multimode information is integrative used to present video contents like subtitles, narrator's voices, on-site sounds, camera movements, scenarios and images etc. It is incomplete to analyze only one mode. For more effective analysis of events, it's required to study the analytical method which fuses multiple patterns. On the other hand, semantic events in those videos are not isolated. There's some logical or consequential relationship among them. In previous paper, we discussed event detection and recognition with the use of the contextual relationship based on dynamic Bayes network. Now on that basis, we'll explore how to fuse multimode information, which is a key issue we're facing here (1-2). In recent years, the fusion of multimode information has become a hot topic in the field of sport video analysis (3-5). Firstly we introduce the related work. Most of the multimode fusion analysis methods mentioned in previous literatures considered the detection of an isolated event. Unlike them, we propose to detect many events and analyze comprehensively the association among them. Multi-level analysis methods based on statistics are built on the probabilistic graphical model, such as Hidden Markov Model (HMM), dynamic Bayes network (DBN) and their variants. By combining visual graphical model representation and effective reasoning and learning methods, such solutions made fairly good effects. Xie (6-7) et al. applied hierarchical HMM for unsupervised clustering to discover layered structure of video contents. Differently, we introduced the method based on dynamic Bayes network model to do the same work. Through learning of training samples, we fulfilled the detection and recognition of wonderful events in the football match (8). Based on DBN, we developed a common sport video analysis framework. Compared with previous work mentioned above, our approach can not only integrate multimode information for event detection and also deal with hierarchical relationship among events. Although the two functions were stated in previous papers, they as a whole were not paid
BibTeX
@article{wang2015research,
  author = {Jia Wang and Haitao Yang and Yang Wang and Jingmeng Sun},
  title = {{Research on Analysis of Sports Video Multi-Pattern Fusion}},
  journal = {International Journal of Database Theory and Application},
  year = {2015},
  doi = {10.14257/ijdta.2015.8.6.28},
}
Figure