SideCow-VSS: A Video Semantic Segmentation Dataset and Benchmark for Intelligent Monitoring of Dairy Cows Health in Smart Ranch Environments

Lei Yao, Jin Liu, Wei Hong, Fanrong Kong, Zipei Fan, Lin Lei, Xinwei LiVeterinary 2025 · DOI

TL;DR

This paper introduces SideCow-VSS, a novel video dataset for semantic segmentation of dairy cows, addressing the critical need for realistic data in precision livestock farming. They also provide a benchmark of eight deep learning models, revealing a clear trade-off between segmentation accuracy and inference speed, with Mask2Former excelling in precision and PIDNet-s in speed.

What Problem Does This Paper Address?

Modern dairy farming is increasingly adopting Precision Livestock Farming (PLF) techniques, leveraging AI and computer vision for automated health monitoring. This allows for early detection of diseases and improved animal welfare. However, a significant bottleneck has been the lack of high-quality, realistic video datasets specifically for dairy cows in their natural farm environments. Existing datasets are often static, lack dense pixel-level annotations, or don't capture the temporal dynamics crucial for monitoring health indicators like gait and body condition. Furthermore, there's a need for standardized benchmarks to evaluate and compare different AI models for this specific task, helping researchers and practitioners choose the right tools for their needs.

Methods

The authors developed a semi-automated pipeline for creating their dataset. This involved:

  1. Initial Object Detection: Using YOLOv11 to generate bounding box proposals for cows in each frame.
  2. Initial Segmentation: Employing the Segment Anything Model 2 (SAM2) to create preliminary segmentation masks based on the bounding boxes.
  3. Manual Verification and Curation: A team of veterinarians and computer vision engineers meticulously reviewed and corrected these masks, discarding frames with significant errors or heavy occlusions.

For the benchmarking, they selected eight diverse semantic segmentation models, ranging from established Convolutional Neural Networks (CNNs) like U-Net and DeepLabV3+ to modern Transformer-based architectures such as SegFormer, SegNeXt-L, and Mask2Former. These models were trained and evaluated under consistent experimental conditions.

Datasets

The paper introduces the SideCow-VSS dataset, a new collection of 921 five-second video clips (totaling 46,050 frames) featuring side-view footage of dairy cows in a commercial farm. These clips are densely annotated at the pixel level. The dataset is split into training (70%), validation (15%), and testing (15%) sets, ensuring that all frames from a single 5-second sequence remain within the same split to prevent data leakage.

Key Insights and Ideas

  • Side-view is Key: The authors emphasize that side-view perspectives are particularly valuable for assessing health indicators like Body Condition Scoring (BCS) and gait, making their dataset uniquely suited for these applications.
  • The "Precision vs. Speed" Dilemma: The benchmark clearly illustrates that achieving the highest segmentation accuracy often comes at the cost of slower inference speeds, and vice-versa. This is a critical consideration for real-world deployment.
  • Transformer Dominance in Accuracy: State-of-the-art Transformer models, particularly Mask2Former with a Swin-L backbone, achieve superior segmentation accuracy, highlighting their ability to capture complex visual features.
  • Lightweight Models for Real-Time: For applications requiring immediate alerts, lightweight CNN-based models like PIDNet-s offer significantly faster inference, making them practical for real-time monitoring.
  • Classic CNNs Remain Relevant: Despite the rise of Transformers, established CNN models like U-Net and DeepLabV3+ still provide robust and competitive accuracy, serving as strong baselines.

Key Results

  • Highest Accuracy: Mask2Former with a Swin-L backbone achieved the highest mean Intersection over Union (mIoU) of 97.32%.
  • Fastest Inference: PIDNet-s model delivered the fastest inference speed of 59.5 Frames Per Second (FPS).
  • Competitive Accuracy: SegNeXt-L achieved an mIoU of 97.00%, while U-Net and DeepLabV3+ scored 96.97% and 96.96% mIoU, respectively.
  • Qualitative Differences: Visual analysis showed Mask2Former providing the most precise boundary delineation, especially around legs and tails, while models optimized for speed sometimes struggled with intricate details.

Limitations and Future Work

  • Dataset Specificity: The SideCow-VSS dataset was collected from a single farm. The authors acknowledge that model performance might vary across different breeds, housing systems, or lighting conditions, suggesting a need for broader environmental diversity in future datasets.
  • Individual Animal Identification: The current semantic segmentation approach cannot distinguish between individual cows when they are in close proximity.
  • Future Directions:
    • Expand the dataset to include more diverse environments.
  • Develop instance segmentation models to enable individual animal tracking for personalized health management.
  • Explore leveraging precise 2D contours for inferring 3D shape information, potentially leading to more advanced health monitoring systems.

Summary updated

Abstract

Accurate and non-invasive monitoring of dairy cows is a cornerstone of precision livestock farming, paving the way for proactive health management and earlier disease detection. The development of robust, AI-driven diagnostic tools, however, is hindered by a dual challenge: scarce realistic video datasets and a lack of standardized benchmarks for deep learning models. To confront these issues, this study puts forward SideCow-VSS, a video semantic segmentation dataset comprising 921 side-view clips with dense, pixel-level annotations of dairy cows under variable on-farm conditions. We systematically evaluated eight deep learning architectures, from classic convolutional neural networks to state-of-the-art Transformers. The evaluation highlighted a clear performance trade-off: the Mask2Former model with a Swin-L backbone yielded the highest mIoU at 97.32%, making it well-suited for detailed morphological analysis. In contrast, the lightweight PIDNet-s model achieved the fastest inference speed of 59.5 FPS, demonstrating its potential for real-time behavioral alerting systems. This work delivers a foundational resource and quantitative framework to inform model selection, accelerating the creation of computer vision systems for automated health monitoring and adopting preventive strategies against key metabolic and immunological disorders in dairy production.
BibTeX
@inproceedings{yao2025sidecow,
  author = {Lei Yao and Jin Liu and Wei Hong and Fanrong Kong and Zipei Fan and Lin Lei and Xinwei Li},
  title = {{SideCow-VSS: A Video Semantic Segmentation Dataset and Benchmark for Intelligent Monitoring of Dairy Cows Health in Smart Ranch Environments}},
  booktitle = {Veterinary},
  year = {2025},
  doi = {10.3390/vetsci12111104},
}
Figure