Video recordings and play behaviour annotations of group-housed dairy calves on two Dutch farms

Haiyu Yang, Heidi Lesscher, Enhong Liu, Miel HostensarXiv 2026 · arXiv

This paper presents a significant new resource for the animal welfare and computer vision communities: a large-scale, open-access dataset focused on the play behavior of dairy calves. By combining hundreds of hours of video with meticulous manual annotations and automated tracking, the authors provide a foundation for developing AI systems that can "see" and interpret positive animal welfare in real-time.

TL;DR

The authors release a massive open-access dataset containing 651 hours of video and 1.1 million annotated frames documenting the play behavior of group-housed dairy calves. This dataset, which includes per-calf tracking and frame-level labels, is designed to help researchers develop computer vision models that can automatically monitor animal welfare through positive behavioral indicators.

What Problem Does This Paper Address?

In the dairy industry, animal welfare has traditionally focused on preventing suffering (disease, hunger, pain). However, modern science is shifting toward "positive welfare"—ensuring animals have good lives, not just lives free of pain. Play behavior is a gold-standard indicator of positive welfare in young calves; they only play when their immediate needs are met and they feel safe.

The challenge is that play is rare—making up only about 1% of a calf's daily time budget—and it is often brief and fast-moving. Measuring it manually is incredibly labor-intensive, and current automated tools like leg-mounted accelerometers lack the visual context to distinguish between different types of play (e.g., social vs. object play). There is a severe lack of high-quality, annotated video data to train the computer vision models needed to solve this.

Methods

The researchers recorded calves on two Dutch farms (one research, one commercial) using ceiling-mounted cameras. Their pipeline combined traditional ethology with modern AI:

  • Manual Annotation: Trained observers used Noldus The Observer XT to code 104 hours of video continuously, identifying 13 different play behaviors (like galloping, bucking, and frontal pushing) and 5 other categories.
  • The Timing Fix: Because farm cameras often drop frames, the researchers couldn't rely on standard video timestamps. They used Optical Character Recognition (OCR) to read the "burned-in" clock on every single frame to ensure the annotations stayed perfectly synced with the video.
  • Computer Vision Pipeline: They used YOLO12x for initial calf detection and the SAMURAI tracker (built on Meta’s Segment Anything Model 2.1) to generate bounding boxes and segmentation contours for individual calves across 1.1 million frames.
  • Label Mapping: Raw behaviors were condensed into a three-class system for machine learning: Active Playing, Non-Active Playing, and Not Playing.

Datasets

The primary contribution is the Calf-Play-Behavior-Dataset, hosted on Hugging Face.

  • Scope: 1,173 video files (651.3 hours) totaling 1.78 TB.
  • Annotations: 104 hours of continuous behavior logs; 90 hours of which have every single calf in the pen coded.
  • Frames: 1,106,110 decoded JPEG frames with associated OCR timestamps, tracking data, and play labels.
  • Subjects: 12 distinct calves across four different recording deployments.

Key Insights and Ideas

  • Play as a "Needle in a Haystack": The data confirms that play is extremely rare, occupying between 0.06% and 1.17% of a calf's time. This highlights why large datasets are necessary; a model needs to see thousands of "normal" moments to accurately identify the rare "play" moments.
  • Social Play Dominates: "Frontal pushing" was the most frequent play behavior recorded (589 events). This suggests that for group-housed calves, social interaction is a primary driver of positive experience.
  • The Hardware Reality Check: The authors' reliance on OCR for timing is a crucial "lesson from the field." It acknowledges that in real-world agricultural settings, hardware isn't perfect, and software must be robust enough to handle dropped frames and variable timing.

Key Results

  • High-Fidelity Labels: The dataset provides frame-level labels for over 57,000 "calf-frames," specifically targeting the most active periods.
  • OCR Success: The OCR system achieved up to 99.8% accuracy in reading timestamps on some sessions, though it struggled in bright daylight glare (dropping to ~3% in the worst-case scenario).
  • Benchmark Ready: By providing segmentation masks and bounding boxes alongside behavior labels, the dataset supports multiple CV tasks: detection, tracking, and action recognition.

Limitations and Future Work

  • Tracking Accuracy: The authors note that tracking was automated and not manually audited for "identity switches" (where the AI confuses one calf for another). Users are encouraged to treat these as "pseudo-labels."
  • Environmental Factors: Daylight glare significantly hampered the OCR timing system in some sessions. Future work might involve more robust time-syncing methods or improved image preprocessing.
  • Diversity: While the dataset is large, it only covers two farms and 12 calves. Expanding this to more diverse environments and breeds would improve model generalization.
  • Sensor Fusion: A promising next step mentioned is using this video "ground truth" to validate and improve the accuracy of cheaper, less-invasive sensors like ear-tag or leg-mounted accelerometers.

Summary updated

Abstract

Play is used as an indicator of positive welfare in dairy calves, but openly available video of group-housed calves with individual behaviour annotations is scarce. We describe video from four camera deployments of about one week each in group pens of dairy calves on two farms in the Netherlands: a research and teaching dairy farm (2024 and 2025) and a commercial dairy farm (2024). The recordings comprise 1,173 video files spanning 651 h. For eight observation days, trained observers coded calf behaviour continuously in Noldus The Observer XT with an 18-item ethogram of play and other behaviours, covering 104 h of video, 90 h of which have every calf in the pen coded. The release also contains 1.1 million decoded frames from seven annotated sessions, a per-frame timestamp read by optical character recognition from the camera clock burned into each frame, per-calf segmentation tracks and per-calf frame-level play labels. The data support studies of calf behaviour and welfare and the development and evaluation of automated behaviour recognition.
BibTeX
@misc{yang2026video,
  author = {Haiyu Yang and Heidi Lesscher and Enhong Liu and Miel Hostens},
  title = {{Video recordings and play behaviour annotations of group-housed dairy calves on two Dutch farms}},
  year = {2026},
  eprint = {2611.21706},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21706},
}
Figure