A dataset of deep learning training for multimodal head images of individual cows

Liang Du, Shanshan Cao, Tingting Liu, Wei Sun, Fantao KongChina 2025 · DOI

Abstract

With the advancement of agricultural modernization and precision agriculture, the intelligent and precision management of livestock farming has shown great potential for applications in areas such as animal health monitoring, farming process management, and precision farming. Leveraging computer vision and infrared imaging, image analysis-based non-contact health monitoring technologies enable real-time monitoring of cows’ physiological states and behavioral characteristics, which can support early disease detection and production management optimization. This study develops a dataset of multimodal head images of individual cows, comprising images collected from 50 cows under natural activity conditions and varying lighting environments (daytime and nighttime). The dataset includes 2,636 pairs of co-registered visible light (VIS) images and infrared thermographic (IRT) images, with precise annotations of cow head regions in the IRT images. Techniques such as blur detection, image quality assessment, and data augmentation were employed to optimize data quality, ensuring diversity and authenticity. This dataset can serve as a foundation for research on cross-modal learning, image pairing, and behavioral analysis, contributing to the development of intelligent monitoring systems and precision agriculture.

Introduction

As the dairy industry scales globally, traditional manual health monitoring has become a significant bottleneck. These methods are often labor-intensive, subjective, and frequently fail to detect early physiological signs of illness. While computer vision (CV) offers a promising non-contact alternative, visible light cameras alone are insufficient for detecting internal physiological changes, such as a rising fever. Conversely, while infrared cameras provide critical thermal data, they often lack the high-resolution spatial context required to identify specific individuals or complex behaviors in crowded environments.

The synergy of multimodal sensing—specifically combining visible light (VIS) and infrared thermography (IRT)—represents the gold standard for automated livestock monitoring. However, the field has been hampered by a lack of publicly available, high-quality, and precisely aligned datasets. This paper introduces a new dataset designed to fill this gap, providing the necessary ground truth to train deep learning models capable of simultaneously understanding a cow's behavior and its physiological state.

Methodology

The researchers established a rigorous pipeline to ensure the dataset's utility for advanced deep learning applications. The process involved five key stages:

  1. Synchronized Data Capture: Utilizing a mobile platform, the team captured 1080p visible light video alongside high-precision thermal video (accurate to ±0.5°C) from 50 individual cows.
  2. Geometric Calibration: To account for lens distortion, standard checkerboard calibration was employed to calculate intrinsic parameters, allowing for the "de-warping" of images.
  3. Co-registration and Alignment: This critical step involved cropping and scaling VIS images to match the 512x384 pixel resolution of the IRT images. By calculating a specific Region of Interest (ROI), the researchers ensured that every pixel in the thermal map corresponds exactly to the same physical point in the visible light photograph.
  4. Quality Control Pipeline: To prevent "garbage in, garbage out" scenarios, the team applied several automated filters, including Laplacian Variance for detecting motion blur, Structural Similarity Index (SSIM) to eliminate redundant frames, and Canny Edge Detection to evaluate noise levels in thermal imagery.
  5. Data Augmentation: To enhance model robustness, the initial 759 high-quality pairs were expanded to a total of 2,636 pairs through random rotation, translation, scaling, and flipping.

Dataset Characteristics

The resulting dataset, titled "A dataset of deep learning training for multimodal head images of individual cows," is hosted on ScienceDB and comprises 1.31 GB of data.

  • Content: 2,636 image pairs (VIS and IRT) with corresponding YOLO-format labels.
  • Structure: The data is organized into training, testing, and validation sets using an 80/10/10 split.
  • Environmental Variety: The dataset includes both daytime and nighttime (infrared-assisted) captures, providing a robust foundation for developing systems that can operate 24/7 in real-world farm environments.

Discussion and Key Insights

A primary focus of this work is the cow's head, which serves as a diagnostic hub. The head contains key temperature monitoring sites, such as the eyes and muzzle, and is the primary location for observing critical behaviors like feeding and rumination.

One of the most striking aspects of the dataset's creation was the intensity of the data pruning process. The authors initially captured over 11,000 frames but discarded approximately 93.63% of them during quality control. This highlights the significant challenges of agricultural computer vision, where environmental noise and motion blur are pervasive. By providing perfectly aligned pairs, this dataset enables research into image fusion (merging thermal and visual data) and cross-modal translation (predicting thermal maps from visual data).

Furthermore, the authors ensured a balanced spatial distribution of annotations. A "label space heatmap" confirms that the head annotations are well-distributed across the frames, which prevents models from developing a "center bias" during the training phase.

Conclusion and Future Work

While this dataset represents a major step forward, several areas for future development remain. Currently, the dataset labels the "head" as a single class. Future iterations could include keypoint detection for specific features like eyes, nostrils, and ears to allow for more granular temperature extraction. Additionally, expanding the scope to include different breeds and more varied housing environments would improve the generalizability of resulting models. Finally, linking physiological thermal changes directly to specific behaviors remains a high-priority goal for the next generation of precision livestock monitoring.

Summary updated

Abstract

<p indent="0mm">With the advancement of agricultural modernization and precision agriculture, the intelligent and precision management of livestock farming has shown great potential for applications in areas such as animal health monitoring, farming process management, and precision farming. Leveraging computer vision and infrared imaging, image analysis-based non-contact health monitoring technologies enable real-time monitoring of cows’ physiological states and behavioral characteristics, which can support early disease detection and production management optimization. This study develops a dataset of multimodal head images of individual cows, comprising images collected from 50 cows under natural activity conditions and varying lighting environments (daytime and nighttime). The dataset includes 2,636 pairs of co-registered visible light (VIS) images and infrared thermographic (IRT) images, with precise annotations of cow head regions in the IRT images. Techniques such as blur detection, image quality assessment, and data augmentation were employed to optimize data quality, ensuring diversity and authenticity. This dataset can serve as a foundation for research on cross-modal learning, image pairing, and behavioral analysis, contributing to the development of intelligent monitoring systems and precision agriculture.
BibTeX
@inproceedings{du2025dataset,
  author = {Liang Du and Shanshan CAO and Tingting Liu and Wei Sun and Fantao Kong},
  title = {{A dataset of deep learning training for multimodal head images of individual cows}},
  booktitle = {China},
  year = {2025},
  doi = {10.11922/11-6035.nasdc.2025.0001.zh},
}
Figure