A wheat spike detection method based on Transformer

Qiong Zhou, Ziliang Huang, Shijian Zheng, Lin Jiao, Liusan Wang, Rujing WangFrontiers in Plant Science 2022 · DOI

TL;DR

Researchers have developed WheatFormer, a new Transformer-based object detection model specifically designed to count wheat spikes in complex field conditions. By combining a multi-window attention mechanism with a specialized "Wheat IoU" loss function, the model achieves state-of-the-art accuracy (0.459 mAP), significantly outperforming traditional convolutional neural networks (CNNs) in dense and occluded environments.

What Problem Does This Paper Address?

Accurately detecting wheat spikes is essential for yield estimation and precision agriculture. However, it is a notoriously difficult computer vision task. Wheat spikes are often densely packed, vary significantly in size and color depending on growth stage, and are frequently occluded by leaves or other spikes.

While CNNs have been the workhorse for this task, they suffer from "tunnel vision"—an inductive bias toward locality. They excel at picking up local textures but struggle to capture the global, long-range dependencies needed to distinguish individual spikes in a crowded, noisy field. Furthermore, standard bounding box regression methods (like standard IoU loss) often fail to provide enough "gradient signal" to accurately align boxes when objects are overlapping or have varying aspect ratios.

Methods

The authors introduce WheatFormer, which integrates two primary innovations into the FCOS (Fully Convolutional One-Stage) detection framework:

  1. Multi-Window Swin Transformer (MW-Swin): Standard Swin Transformers use fixed-size windows for self-attention. WheatFormer introduces a multi-window strategy (using window sizes of 7, 9, and 11) to extract multi-scale features. This allows the model to adapt to spikes of different sizes and distances from the camera more effectively than a single-scale approach.
  2. Wheat Intersection over Union (WIoU) Loss: Bounding box accuracy is critical for counting. The authors proposed WIoU, which improves upon standard IoU by adding penalty terms that account for the Euclidean distance between box centers and the consistency of aspect ratios. This helps the model "pull" predicted boxes toward the ground truth more aggressively during training.

The model was released in three sizes—Small, Base, and Large—to allow for trade-offs between computational efficiency and raw accuracy.

Datasets

The study utilizes a newly constructed dataset called WSD-2022, which contains a total of 6,404 images.

  • New Data: 978 images were manually collected by the authors across four locations in China, covering various growth stages (flowering to milk stage) and shooting angles.
  • Existing Data: 5,426 images were integrated from the well-known Global Wheat Head Detection (GWHD) dataset.
  • Diversity: The dataset includes variations in color, density, and lighting to ensure the model can generalize to real-world agricultural settings.

Key Insights and Ideas

  • Breaking the Local Bias: The transition from CNNs to Transformers in agriculture isn't just a trend; it’s a functional necessity. The Transformer’s global receptive field allows the model to "understand" the structure of the entire plant, helping it distinguish a spike from a similarly textured leaf.
  • Scale Matters: By using multiple window sizes in the attention mechanism, the authors essentially created a "feature pyramid" within the Transformer blocks themselves, rather than just at the end of the backbone.
  • Geometry-Aware Loss: The "Aha!" moment for the loss function was realizing that standard IoU is "blind" when boxes don't overlap. By incorporating Euclidean distance and aspect ratio, the model receives a constant stream of feedback on how to improve the box shape, even during early training phases.

Key Results

WheatFormer outperformed several heavyweights in the object detection world, including Faster R-CNN, YOLOv3, and SSD.

  • Accuracy: The Base model achieved a mean Average Precision (mAP) of 0.459 and an AP50 of 0.918.
  • Comparison: Compared to the standard Swin Transformer-based FCOS, WheatFormer improved mAP by 0.7% and AP50 by 3.2%.
  • Qualitative: Visualizations show that while models like YOLOv3 missed many small spikes and Faster R-CNN produced too many overlapping boxes, WheatFormer maintained high precision even in high-density areas with significant occlusion.

Limitations and Future Work

Despite the strong results, the authors highlight a few areas for improvement:

  • Model Complexity: The parameters for the "Base" and "Large" versions are relatively high (60.1MB and 100.6MB respectively), which might make deployment on edge devices (like drones or handheld sensors) challenging.
  • Dataset Size: While 6,400 images is a good start, the authors aim to collect more data across even more diverse regions and growth cycles to further harden the model against "out-of-distribution" field conditions.
  • Efficiency: Future work will focus on designing more lightweight versions of the Transformer architecture to improve real-time performance in the field.

Summary updated

Abstract

Wheat spike detection has important research significance for production estimation and crop field management. With the development of deep learning-based algorithms, researchers tend to solve the detection task by convolutional neural networks (CNNs). However, traditional CNNs equip with the inductive bias of locality and scale-invariance, which makes it hard to extract global and long-range dependency. In this paper, we propose a Transformer-based network named Multi-Window Swin Transformer (MW-Swin Transformer). Technically, MW-Swin Transformer introduces the ability of feature pyramid network to extract multi-scale features and inherits the characteristic of Swin Transformer that performs self-attention mechanism by window strategy. Moreover, bounding box regression is a crucial step in detection. We propose a Wheat Intersection over Union loss by incorporating the Euclidean distance, area overlapping, and aspect ratio, thereby leading to better detection accuracy. We merge the proposed network and regression loss into a popular detection architecture, fully convolutional one-stage object detection, and name the unified model WheatFormer. Finally, we construct a wheat spike detection dataset (WSD-2022) to evaluate the performance of the proposed methods. The experimental results show that the proposed network outperforms those state-of-the-art algorithms with 0.459 mAP (mean average precision) and 0.918 AP 50 . It has been proved that our Transformer-based method is effective to handle wheat spike detection under complex field conditions.
BibTeX
@inproceedings{zhou2022wheat,
  author = {Qiong Zhou and Ziliang Huang and Shijian Zheng and Lin Jiao and Liusan Wang and Rujing Wang},
  title = {{A wheat spike detection method based on Transformer}},
  booktitle = {Frontiers in Plant Science},
  year = {2022},
  doi = {10.3389/fpls.2022.1023924},
}
Figure