MmCows: A Multimodal Dataset for Dairy Cattle Monitoring

Hien Vu, Omkar Prabhune, Unmesh Raskar, Dimuth Panditharatne, Hanwook Chung, Christopher Y. Choi, Younghyun KimNeurIPS 2024

Introduction

Precision Livestock Farming (PLF) aims to use technology to monitor individual animals in real-time, allowing farmers to intervene early when a cow is sick, stressed, or in heat. While we have plenty of sensors—like GPS for location or accelerometers for movement—most existing datasets are "single-modality." They might provide video or sensor data, but rarely both simultaneously for the same animal.

This lack of synchronized data makes it difficult to develop robust AI. For example, a camera might see a cow standing still, but it cannot determine her internal body temperature. Conversely, a temperature sensor might flag a fever, but without location data, you don't know if she has isolated herself from the herd. MmCows bridges this gap by providing a comprehensive, multi-perspective view of cattle life. It is the first large-scale, high-quality multimodal dataset designed to revolutionize dairy cattle monitoring by synchronizing data from wearable sensors, internal implants, environmental logs, and high-resolution video.

Experimental Setup and Methods

The researchers deployed a sophisticated sensor network at the University of Wisconsin–Madison’s Agricultural Research Station.

Multimodal sensor setup and multi-view camera feeds

The comprehensive setup included:

  • Custom Neck-Mount Tags: These "smart collars" featured Ultra-Wideband (UWB) for precise 3D localization, an Inertial and Magnetic Measurement Unit (IMMU) for head movement, and a pressure sensor for elevation.
  • Implantable Sensors: Ten cows were equipped with vaginal temperature loggers to monitor Core Body Temperature (CBT)—the "gold standard" for detecting heat stress.
  • Ankle Sensors: Used specifically to track lying versus standing behavior.
  • Computer Vision: Four GoPro cameras captured 4.5K video at 1 fps from isometric views, providing 4.8 million frames of data.
  • Environmental & Production Data: The team logged indoor/outdoor weather (to calculate the Temperature-Humidity Index, or THI) and daily milk yields.

To ensure all this data lined up, a stationary hub synchronized every sensor to Internet time every 30 minutes. For ground truth, the team manually annotated 20,000 frames with 213,000 bounding boxes and used an optimization-based approach (AdaGrad) to project 2D camera coordinates into 3D world coordinates.

The MmCows Dataset

The primary contribution is the MmCows dataset itself, which stands out significantly from previous benchmarks. It encompasses 14 days of data from 16 Holstein cows across 9 distinct data streams (UWB, IMMU, Pressure, CBT, Ankle, RGB Video, Indoor THI, Outdoor Weather, Milk Weight). Unlike existing cattle datasets that focus only on RGB video or a single sensor type, MmCows is the first to integrate physiological (CBT) and behavioral data at this scale.

Key Insights and Results

The team benchmarked several machine learning models (Random Forests for sensors, YOLOv8/EfficientNet for vision) on behavior classification, revealing several key insights:

  • The Power of Sensor Fusion: While UWB (location) is great for knowing where a cow is, it struggles to distinguish between similar behaviors like standing versus drinking. Adding Head Direction (HD) and Ankle data significantly reduces errors by providing necessary context for the cow's posture. The multimodal fusion of UWB + HD + Ankle achieved the highest F1 scores, particularly for "Lying" (0.997) and "Standing" (0.938).
  • Vision vs. Sensors: Vision-only models (RGB) are excellent for identification but struggle with dynamic actions like "walking" versus "standing" because they often evaluate individual frames. Sensors, which capture temporal patterns better, outperformed vision in several behavior categories.
  • Multi-View Vision Improvements: Using a multi-view approach (RGBm) significantly improved performance over single-camera views (RGBs), raising the average F1 score from 0.554 to 0.632.
  • Heat Stress Correlation: The data confirmed a strong correlation between the Temperature-Humidity Index (THI) and cow behavior (p-values < 0.05). As THI rises, cows stand more (to cool down via convection) and their milk yield drops—a vital insight for climate-resilient farming and health monitoring.

Limitations and Future Work

While MmCows is a massive step forward, the authors note a few areas for future growth:

  • Sample Size: The dataset tracks 16 cows (10 with full sensor suites). While high-quality, larger herds would provide even more robust statistical power.
  • Temporal Vision: The current vision benchmarks use individual frames. Using video sequences (temporal data) would likely improve "walking" detection, which currently has a lower F1 score.
  • Generalization: The data was collected over two weeks in the summer. Future iterations could include different seasons or barn layouts to ensure models do not overfit to one specific environment.

Summary updated

BibTeX
@inproceedings{vu2024mmcows,
  author = {Hien Vu and Omkar Prabhune and Unmesh Raskar and Dimuth Panditharatne and Hanwook Chung and Christopher Y. Choi and Younghyun Kim},
  title = {{MmCows: A Multimodal Dataset for Dairy Cattle Monitoring}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
}
Figure