Unsupervised Few Shot Key Frame Extraction for Cow Teat Videos

Youshan Zhang, Matthias Wieland, Parminder S. Basran

TL;DR

This paper introduces a novel unsupervised few-shot learning approach (UFSKFE) to automatically extract key frames from dairy cow teat videos. This is crucial for efficient health monitoring in large-scale dairy farms, where manual annotation is impractical. The method achieves a competitive F score of 63.6% with a processing time of under 10 minutes per video, making it suitable for real-world applications.

What Problem Does This Paper Address?

Monitoring the health of dairy cows, particularly their teats, is vital for milk production quality and disease prevention (like mastitis). In large commercial dairy farms, this involves inspecting thousands of cows, a task that is time-consuming and expensive. While video analysis offers a potential solution, processing entire video streams frame-by-frame is inefficient. Key frame extraction (KFE) aims to identify the most representative frames, but existing methods face challenges in this specific domain. Traditional supervised KFE requires extensive manual labeling, which is infeasible for large datasets. Unsupervised methods often rely on significant frame changes, which are not present in cow teat videos where frames can be quite similar. Furthermore, the data collection environment is harsh, leading to unpredictable video quality (e.g., obscured teats, empty stalls, camera reversals).

Methods

The authors propose a novel unsupervised few-shot learning key frame extraction (UFSKFE) model. The core idea is to leverage a small set of pre-defined "support" key frames (KFs) to guide the extraction of KFs from unlabeled video data.

Here's a breakdown of their approach:

  • Fusion Distance: They combine two types of distances to represent the similarity between each video frame and the support KFs:
  • Raw Distance: This uses the L1 norm (Manhattan distance) between the raw RGB pixel values of video frames and support KFs.
  • Deep Distance: This leverages deep features extracted from a pre-trained ResNet-101 model (trained on ImageNet) to capture more abstract visual information. The L1 norm is also used here.
  • These two distances are then combined using a weighted sum, controlled by a hyperparameter $\alpha$, to form a robust fusion distance.
  • Key Frame Selection Mechanism: Instead of a simple threshold, they employ a mechanism that sorts the fusion distances and iteratively selects the frame with the smallest distance. To avoid redundancy, they then remove a window of frames around the selected KF.
  • Quality Check (QC): A crucial step to filter out noisy or irrelevant KFs. This involves calculating the Structural Similarity Index (SSIM) between the detected KF and its closest support KF within a fixed teat region. If the SSIM score falls below a threshold, the KF is discarded.

Datasets

The study utilizes a newly collected dataset of dairy cow teat videos.

  • CowTeatVideo Benchmark: This is a new, publicly available dataset consisting of dairy cow teat videos specifically for KFE research.
  • Data Collection: Videos were recorded using a GoPro 10 camera with LED lights on a commercial dairy farm housing approximately 1600 Holstein cows. The data comprises about eight hours of footage, with videos sampled at 1080x1920 resolution and 59.94 fps. The dataset contains 18 videos, with an average of 21,191 frames per video. The number of key frames per video was manually checked by an expert.

Key Insights and Ideas

  • Few-Shot Learning for KFE: The paper demonstrates that few-shot learning, typically used for classification, can be effectively adapted for unsupervised key frame extraction, even with very limited labeled data (only 32 support KFs).
  • Fusion of Raw and Deep Features: Combining simple raw pixel distances with sophisticated deep features proves more effective than using either alone. This suggests that both low-level and high-level visual cues are important for identifying relevant cow teat frames.
  • Importance of Quality Check: The SSIM-based quality check is critical for filtering out frames where the milking apparatus obstructs the view of the teats, significantly improving the accuracy of the extracted KFs.
  • Unsupervised Approach is Key: The success of an unsupervised method is vital for practical application in large-scale dairy farms, where manual annotation is a major bottleneck.

Key Results

  • High F Score: The proposed UFSKFE model achieves the highest average F score (63.6%) compared to various baseline methods that use different feature extraction techniques (e.g., SURF, AlexNet, NASNetLarge) and distance metrics.
  • Efficient Processing: The model can extract key frames from a video with approximately 21,000 frames in under 10 minutes, which is crucial for real-time monitoring in commercial settings.
  • Ablation Study: An ablation study confirms the effectiveness of each component of the UFSKFE model. The fusion distance (d) performs better than raw (draw) or deep (ddeep) distances alone. Adding the quality check (+ QC) further improves performance.

Limitations and Future Work

  • Milking Apparatus Obstruction: The model still struggles to completely remove KFs where the milking apparatus is attached to the cow's teats, especially if the camera's field of view is limited. The authors suggest repositioning the camera (portrait vs. landscape) as a potential workaround.
  • Single Key Frame per Cow: The current approach extracts only one key frame per cow. Future work could involve extracting key frames for each individual teat, providing more detailed information.
  • Other Feature Extraction Methods: The authors suggest exploring other feature extraction methods from few-shot learning to potentially improve performance.
  • Real-time Cloud Processing: The possibility of performing this process in real-time by storing recorded videos on a cloud console is mentioned.
  • Assessing Hyperkeratosis: Future work will focus on using the extracted key frames to assess the extent of hyperkeratosis and the risk of mastitis, moving towards a complete automated health monitoring system.

Summary updated

Abstract

A novel method of monitoring the health of dairy cows in large-scale dairy farms is proposed via image-based analysis of cows on rotary-based milking platforms, where deep learning is used to classify the extent of teat-end hyperkeratosis. The videos can be analyzed to segment the teats for feature analysis, which can then be used to assess the risk of infections and other diseases. This analysis can be performed more efficiently by using the key frames of each cow as they pass through the image frame. Extracting key frames from these videos would greatly simplify this analysis, but there are several challenges. First, data collection in the farm setting is harsh, resulting in unpredictable temporal key frame positions; empty, obfuscated, or shifted images of the cow’s teats; frequently empty stalls due to challenges with herding cows into the parlor; and regular interruptions and reversals in the direction of the parlor. Second, supervised learning requires expensive and time-consuming human annotation of key frames, which is impractical in large commercial dairy farms housing thousands of cows. Unsupervised learning methods rely on large frame differences and often suffer low performance. In this paper, we propose a novel unsupervised few-shot learning model which extracts key frames from large (∼21,000 frames) video streams. Using a simple L1 distance metric that combines both image and deep features between each unlabeled frame and a few (32) labeled key frames, a key frame selection mechanism, and a quality check process, key frames can be extracted with sufficient accuracy (F score 63.6%) and timeliness (<10 min per 21,000 frames) for commercial dairy farm setting demands.
BibTeX
@article{zhang2022unsupervised,
  author = {Youshan Zhang and Matthias Wieland and Parminder S. Basran},
  title = {{Unsupervised Few Shot Key Frame Extraction for Cow Teat Videos}},
  journal = {Data},
  year = {2022},
  doi = {10.3390/data7050068},
}
Figure