Self-Supervised Shape Alignment for Sports Field Registration

Feng Shi, Paul Marchwica, Juan Camilo Gamboa Higuera, Michael Jamieson, Mehrsan Javan, Parthipan SivaWACV 2022

TL;DR

This paper introduces a novel self-supervised, end-to-end deep learning approach for cross-modality image registration, specifically for aligning sports field templates with broadcast video frames. By leveraging a score regression network for iterative refinement and a clever data augmentation strategy, the method achieves state-of-the-art results in sports field registration without requiring any manual annotations.

What Problem Does This Paper Address?

Registering images from different modalities (e.g., a natural image to an edge map or a template) is a challenging computer vision problem. Traditional methods often struggle with the lack of texture and color information in templates, and existing deep learning approaches have primarily focused on single-modality registration or have been biased towards texture-rich natural images. Sports field registration, a specific application of cross-modality registration, is crucial for augmented reality overlays, player tracking, and game analysis in sports broadcasts. Existing methods for this task often rely on complex pipelines, manual feature engineering, or require significant training data. This paper aims to bridge this gap by proposing a self-supervised, end-to-end deep learning solution that can effectively learn shape-based correspondences between a natural image and a template, even when texture information is absent.

Methods

The core of the proposed method is a novel end-to-end self-supervised learning framework for homography estimation. Instead of relying on pairwise labeled data, the authors train a regression network using natural images and their corresponding edge maps. A key innovation is the use of a Score Regression Network (SRN). This SRN estimates the alignment error, allowing the system to iteratively refine the homography estimate. This iterative process, combined with image warping, enables the network to converge to an accurate alignment even with large initial misalignments.

The training strategy is also a significant contribution. The authors propose a self-supervised data mining method that generates training data by perturbing ground truth homographies and warping edge maps. This approach avoids the need for manual annotation and allows the network to learn robust shape representations. They also explore two data augmentation strategies to create diverse training samples, ensuring the network generalizes well. For the network architecture, they utilize a ResNet backbone, with specialized shallow processing streams for the edge input to handle its sparse nature.

Datasets

The paper primarily uses the ImageNet dataset for training and evaluation of the general homography estimation task. For the specific application of sports field registration, they evaluate their method on the WorldCup soccer dataset, the Hockey dataset, and the Volleyball dataset. These datasets are commonly used benchmarks for sports field registration tasks. The authors note that their method can achieve state-of-the-art results even when using a significantly smaller subset of the Hockey dataset compared to prior work.

Key Insights and Ideas

  • Self-supervised learning for cross-modality registration: The paper demonstrates that it's possible to train a robust homography estimation network for cross-modality tasks (like image-to-edge map) without any manual labels, relying solely on the inherent structure of the data and clever data augmentation.
  • The power of iterative refinement with a score estimator: The SRN is crucial. It not only guides the refinement process but also allows for adaptive stopping, balancing accuracy and speed. This is a significant improvement over methods that require a fixed, often large, number of iterations.
  • Shape-based learning is key for template matching: The success of the method on sports fields, which are essentially shape templates, highlights the importance of learning shape features when texture is absent. This is a critical insight for applications where traditional feature descriptors might fail.
  • Pretraining significantly boosts performance: Fine-tuning the homography network with weights from a pre-trained ImageNet model (or even a self-trained homography model) leads to substantial improvements, demonstrating the effectiveness of transfer learning in this domain.

Key Results

  • The proposed method achieves state-of-the-art results on sports field registration benchmarks, outperforming existing methods in terms of mean and median Average Corner Errors.
  • On the soccer dataset, the method shows a 2.66 increase in mean IoUwhole and a 1.51 increase in mean IoUpart compared to prior work.
  • The iterative refinement process, guided by the SRN, significantly reduces the number of required iterations for accurate alignment. For example, they show that their SRN can reduce the iterations from 400 to around 299.4 on a challenging dataset while maintaining accuracy.
  • The method is capable of real-time processing, achieving over 100fps on an Nvidia Tesla T4 GPU using FP16.
  • A hybrid approach, combining their network with traditional feature detectors like AKAZE, yields very good results, suggesting a promising direction for robust registration.

Limitations and Future Work

  • Failed cases in symmetrical scenarios: The paper acknowledges that in cases with very little overlap and high symmetry (e.g., the "Boston Library" example in Figure 7), the network might converge to a local maximum, leading to a failed alignment.
  • Potential for further optimization: While the method is already fast, there's always room for further optimization, especially for real-time applications on less powerful hardware.
  • Generalizability to other cross-modality tasks: While demonstrated on sports fields, the authors suggest their method is not limited to cross-modality registration and can be applied to image-to-image tasks. Future work could explore its application in other domains requiring shape-based alignment.
  • Exploration of different backbone architectures: The paper uses ResNet, but exploring other CNN architectures could potentially yield further improvements.

Summary updated

BibTeX
@inproceedings{shi2022self,
  author = {Feng Shi and Paul Marchwica and Juan Camilo Gamboa Higuera and Michael Jamieson and Mehrsan Javan and Parthipan Siva},
  title = {{Self-Supervised Shape Alignment for Sports Field Registration}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure