A comparative study of Fourier transform and CycleGAN as domain adaptation techniques for weed segmentation

Riccardo Bertoglio, Alessio Mazzucchelli, Nico Catalano, Matteo MatteucciSmart 2023 · DOI

TL;DR

This paper tackles the challenge of weed segmentation in precision agriculture, where models trained on one set of conditions often fail on others due to variations in cameras, lighting, and plant growth. The authors compare two domain adaptation techniques, Fourier Transform and CycleGAN, demonstrating that CycleGAN excels when the domain shift is due to different camera hardware, while both struggle when the shift is due to different plant growth stages over time.

What Problem Does This Paper Address?

Precision agriculture aims to optimize farming practices by responding to field variability. A key task is identifying and segmenting weeds to enable targeted herbicide application or mechanical removal, thereby reducing chemical use and environmental impact. Convolutional Neural Networks (CNNs) are powerful for this, but they require large amounts of labeled data. The agricultural environment is highly dynamic, with variations in lighting, soil, weed types, and plant growth stages. This variability creates a "domain gap" between the data a model is trained on (source domain) and the data it encounters in the field (target domain), leading to a significant drop in performance. Domain Adaptation (DA) techniques aim to bridge this gap, reducing the need for extensive re-labeling for every new scenario. This paper specifically investigates unsupervised DA methods that don't require labels from the target domain.

Methods

The study compares two main unsupervised domain adaptation techniques for weed segmentation:

  • Fourier Transform: This method leverages the property that an image's amplitude spectrum contains style information, while its phase spectrum contains semantic information. By swapping the amplitude spectrum of a source image with that of a target image, the goal is to transfer the target's style while preserving the source's semantic content (i.e., weed and crop identification).
  • CycleGAN: This is a generative adversarial network (GAN) architecture designed for unpaired image-to-image translation. It uses two generators and two discriminators to translate images between two domains (e.g., source to target, and target to source). To improve semantic consistency (ensuring that translated images still represent the same objects), the authors explore two modifications:
  • CGAN L_semantic: Incorporates losses based on Intersection over Union (IoU) to ensure that the predicted segmentation masks of the original and translated images are similar.
  • CGAN L_phase: Adds a loss term that penalizes the phase discrepancy between real and transformed images, further enforcing semantic preservation.

These methods are compared against a baseline (model trained on source, tested on target without DA) and an upper bound (model trained and tested on target with labels). The segmentation model used is a U-Net architecture.

Datasets

The research utilizes data from the ROSE Challenge, a competition focused on autonomous weed destruction. The dataset comprises RGB images and corresponding semantic segmentation masks for distinguishing crop, weed, and background classes. The data was collected from experimental fields in Montoldre, France, during 2019 and 2021, featuring maize and bean crops with four common weed species.

The study specifically creates two experimental settings to simulate different domain shifts:

  1. Different Robots/Cameras: Images collected by different teams (BIPBIP and WeedElec) in the same period (May 2019) using distinct robot and camera systems.
  2. Different Years/Growth Stages: Images collected by the same robot (BIPBIP) in different years (2019 and 2021), capturing plants at varying growth stages.

The authors also release a new labeled dataset and the code for their experiments.

Key Insights and Ideas

  • Camera vs. Growth Stage Matters: The effectiveness of DA techniques is highly dependent on the nature of the domain shift. CycleGAN-based methods perform well when the shift is due to different camera hardware, suggesting that generative models can effectively learn to translate visual styles.
  • Phase Matters for Semantics: Introducing a phase-discrepancy loss in CycleGAN (CGAN L_phase) not only stabilizes training but also improves performance, highlighting the critical role of phase information in preserving semantic content during image translation.
  • Limitations of Pixel-Level DA for Temporal Shifts: When the domain shift is due to changes in plant appearance over time (different growth stages), pixel-level DA methods like Fourier Transform and CycleGAN struggle to outperform a baseline model trained without DA. This suggests that feature-level DA might be more suitable for such temporal variations.
  • Artifacts and Over-segmentation: Both Fourier Transform and CycleGAN methods can introduce visual artifacts in transformed images. Furthermore, CycleGAN models sometimes tend to over-segment weeds, identifying more weeds than are present in the ground truth.

Key Results

  • Different Cameras: In scenarios with domain shifts caused by different camera hardware, CycleGAN (especially with the semantic loss) generally outperformed the Fourier Transform and the baseline. For instance, in Table 1, CGAN L_semantic achieved mean IoU scores of 0.827 and 0.844 in two of the four "different robots" comparisons, significantly higher than the baseline's 0.741 and 0.726.
  • Different Years/Growth Stages: For domain shifts due to different growth stages across years, all tested DA techniques struggled. In Table 2, the baseline often performed comparably or better than the DA methods. For example, in the "Bean, 2019 -> 2021" scenario, the baseline achieved a mean IoU of 0.807, while CGAN L_semantic and CGAN L_phase achieved 0.426 and 0.628, respectively.
  • CGAN L_phase Stability: The addition of the phase loss (CGAN L_phase) consistently led to more stable training and improved mean IoU compared to CGAN L_semantic in several experiments, particularly evident in the maize experiments (as shown in Figures 7 and 8).

Limitations and Future Work

  • Temporal Domain Shifts: The paper acknowledges that none of the tested DA techniques consistently beat the baseline when the domain shift is due to different plant growth stages over time. This suggests that pixel-level DA might not be sufficient for such challenges.
  • Class Imbalance: In some experimental settings, the proportion of background pixels (soil) was significantly higher than crop or weed pixels, leading to poorer discrimination for the latter. The authors suggest that a weighted IoU loss could help address this.
  • Future Directions: The authors propose that feature-level DA methods might be more effective for temporal domain shifts. They also emphasize the need for further research to overcome the challenges posed by varying plant growth stages and class imbalances. The release of their dataset and code aims to facilitate future research in this area.

Summary updated

Abstract

Automatic weed identification is becoming increasingly important in the Precision Agriculture field as a fundamental capability for targeted spraying or mechanical weed destruction. Targeted weed elimination reduces herbicides' use and thus lowers the environmental impact of treatments. Convolutional Neural Networks are one of the most successful techniques to automatically detect weeds on RGB images. Such models require a high amount of labeled data to obtain satisfying detection performance. The agricultural context presents a high degree of variability, and it is thus unfeasible to expect a representative dataset for each specific condition that can appear in the fields. Domain Adaptation techniques are exploited to maintain high detection performance in different field conditions, lowering the need for labeled data. This study presents a comparison of the two main style transfer techniques for performing domain adaptation, that is, the Fourier Transform and the CycleGAN architecture. We used these techniques to reduce the domain gap in two use cases: one with images collected by different robots with different cameras and another with images collected by the same platform in different years. We show how, in the first case, the CycleGAN architecture attains satisfying performance and beats the simpler Fourier Transform. Instead, in the second case, all the tested DA techniques struggle to reach baseline performance. We also show how introducing a loss based on phase discrepancy in the CycleGAN architecture stabilizes the training and improves the performance. Moreover, we release a new dataset of labeled agricultural images and the code of our experiments for the reproducibility of the results and comparison with future works.
BibTeX
@inproceedings{bertoglio2023comparative,
  author = {Riccardo Bertoglio and Alessio Mazzucchelli and Nico Catalano and Matteo Matteucci},
  title = {{A comparative study of Fourier transform and CycleGAN as domain adaptation techniques for weed segmentation}},
  booktitle = {Smart},
  year = {2023},
  doi = {10.1016/j.atech.2023.100188},
}
Figure