A Novel Dataset for Evaluating and Alleviating Domain Shift for Human Detection in Agricultural Fields

Paraskevi Nousi, Emmanouil Mpampis, Nikolaos Passalis, Ole Green, Anastasios TefasarXiv 2022 · arXiv

TL;DR

This paper introduces the OpenDR Humans in Field dataset to address the "domain shift" problem where standard human detectors fail in agricultural settings. The authors demonstrate that while pretrained models struggle with false positives in fields, fine-tuning them with even a small amount of target-domain data—including unannotated background images—significantly boosts reliability and safety for agricultural robotics.

What Problem Does This Paper Address?

Object detection models are typically trained on massive, diverse datasets like MS COCO or PASCAL VOC. While these models perform exceptionally well in urban environments, they often stumble when deployed in specialized settings like agricultural fields. This phenomenon is known as domain shift.

In a field, a robot encounters unique challenges:

  • Visual Backgrounds: Mud, crops, and specialized machinery look nothing like the city streets or indoor scenes found in standard datasets.
  • Sensor Characteristics: Agricultural robots often use wide-angle lenses, which distort human proportions compared to standard photography.
  • Safety Stakes: In human-robot interaction (HRI) within farming, a false negative (missing a person) is a safety hazard, while frequent false positives (detecting a tractor part as a person) lead to inefficient "crying wolf" scenarios that hinder autonomy.
Fig. 1: Examples of detections using a pretrained model. Existing models lead to false detections under distribution shifts.

Methods

The researchers evaluated several popular single-stage detector families, including SSD, YOLOv3, and CenterNet. To bridge the domain gap, they proposed and tested three fine-tuning strategies:

  1. Negative-only training: Using only background images (images without humans). This is attractive because it requires zero manual annotation.
  2. Positive-only training: Using only images containing humans.
  3. Balanced Source + Target training: To prevent "catastrophic forgetting" (where a model forgets how to detect humans in general while learning the new field environment), they mixed the target field data with the 'person' class subset from the original COCO dataset.

Datasets

The core contribution is the OpenDR Humans in Field dataset, collected using the "Robotti" agricultural platform.

  • Total Images: 8,038 images.
  • Annotated Set: 818 images (158 positive with bounding boxes, 647 negative background samples).
  • Unsupervised Set: 7,233 images provided for future research into unsupervised domain adaptation.
  • Comparison Sets: The models were originally trained on PASCAL VOC (20 classes) and MS COCO (80 classes).
Fig. 2: Examples of collected images: (a) Image depicting two humans, annotated with bounding boxes, (b) Image depicting no humans.

Key Insights and Ideas

  • The Power of Negatives: One of the most practical findings is that fine-tuning with only negative (background) samples significantly reduces the false positive rate. This suggests that developers can improve model robustness simply by letting a robot drive around a field and collect "empty" footage.
  • Localization Requires Positives: While negative samples stop the model from seeing ghosts, positive samples (images with people) are essential for precise localization (getting the bounding box right).
  • Source Mixing is Best: The best results came from mixing the new field data with original COCO data. This "balanced" approach achieved the highest precision while ensuring the model didn't overfit to the specific people or lighting conditions in the small target dataset.

Key Results

The baseline models showed a significant performance drop when tested on the full field dataset. For example, a standard SSD trained on COCO dropped from an initial precision of 80.3% (on positive-only samples) to 70.1% when background images were included, due to false positives.

By applying the COCO+Both (Source + Target) fine-tuning strategy:

  • Precision at 0.5 IoU jumped from 70.1% to 93.9%.
  • Recall improved from 55.4% to 70.1%.
  • Qualitative Improvement: The model successfully ignored robot chassis parts and irrigation equipment that previously triggered false detections.
Fig. 3: Examples of detections using our COCO+Both model: (a) Image depicting two humans, captured by the front camera of the robot, (b) Image depicting one human, captured by the back camera of the robot.

Limitations and Future Work

The authors acknowledge that the annotated portion of the dataset is relatively small (818 images). While the results are promising, the model's performance might vary under more diverse weather conditions or different crop stages not captured in the current batches.

Future work points toward utilizing the larger, unannotated set of 7,233 images through unsupervised domain adaptation techniques, which could further reduce the need for costly manual labeling while maintaining high safety standards for agricultural robotics.

Summary updated

Abstract

In this paper we evaluate the impact of domain shift on human detection models trained on well known object detection datasets when deployed on data outside the distribution of the training set, as well as propose methods to alleviate such phenomena based on the available annotations from the target domain. Specifically, we introduce the OpenDR Humans in Field dataset, collected in the context of agricultural robotics applications, using the Robotti platform, allowing for quantitatively measuring the impact of domain shift in such applications. Furthermore, we examine the importance of manual annotation by evaluating three distinct scenarios concerning the training data: a) only negative samples, i.e., no depicted humans, b) only positive samples, i.e., only images which contain humans, and c) both negative and positive samples. Our results indicate that good performance can be achieved even when using only negative samples, if additional consideration is given to the training process. We also find that positive samples increase performance especially in terms of better localization. The dataset is publicly available for download at https://github.com/opendr-eu/datasets.
BibTeX
@misc{nousi2022novel,
  author = {Paraskevi Nousi and Emmanouil Mpampis and Nikolaos Passalis and Ole Green and Anastasios Tefas},
  title = {{A Novel Dataset for Evaluating and Alleviating Domain Shift for Human Detection in Agricultural Fields}},
  year = {2022},
  eprint = {2209.13202},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2209.13202},
}
Figure