Doing More With Less: A Multitask Deep Learning Approach in Plant Phenotyping

Andrei Dobrescu, Mario Valerio Giuffrida, Sotirios A. TsaftarisFrontiers in Plant Science 2020 · DOI

Abstract

Image-based plant phenotyping has been steadily growing and this has steeply increased the need for more efficient image analysis techniques capable of evaluating multiple plant traits. Deep learning has shown its potential in a multitude of visual tasks in plant phenotyping, such as segmentation and counting. Here, we show how different phenotyping traits can be extracted simultaneously from plant images, using multitask learning (MTL). MTL leverages information contained in the training images of related tasks to improve overall generalization and learns models with fewer labels. We present a multitask deep learning framework for plant phenotyping, able to infer three traits simultaneously: (i) leaf count, (ii) projected leaf area (PLA), and (iii) genotype classification. We adopted a modified pretrained ResNet50 as a feature extractor, trained end-to-end to predict multiple traits. We also leverage MTL to show that through learning from more easily obtainable annotations (such as PLA and genotype) we can predict a better leaf count (harder to obtain annotation). We evaluate our findings on several publicly available datasets of top-view images of Arabidopsis thaliana. Experimental results show that the proposed MTL method improves the leaf count mean squared error (MSE) by more than 40%, compared to a single task network on the same dataset. We also show that our MTL framework can be trained with up to 75% fewer leaf count annotations without significantly impacting performance, whereas a single task model shows a steady decline when fewer annotations are available.

Introduction

In plant breeding and precision agriculture, "phenotyping"—the measurement of physical traits like leaf size, count, or growth rate—is essential for identifying high-performing crop varieties. While image-based phenotyping offers a faster alternative to manual measurement, it faces two significant hurdles. First, task complexity: leaf counting is notoriously difficult due to overlapping leaves and varying shapes. Second, the annotation bottleneck: training deep learning models typically requires massive amounts of manually labeled data. Counting leaves in thousands of images is a tedious, error-prone process that slows research progress.

Overview of the multitask learning framework for plant phenotyping.

This research addresses these challenges by employing Multitask Learning (MTL). The core hypothesis is that "easier" information, such as the total area a plant covers (Projected Leaf Area), can help a model learn "harder" information, such as the exact number of leaves, by providing structural context and regularizing the learning process.

Methods and Architecture

The proposed framework utilizes a unified network based on a modified ResNet50 architecture. Instead of building three separate models for three different traits, the authors designed a single network that shares early feature extraction layers.

  • Feature Extraction: The model uses the initial layers of ResNet50 to extract high-level features from RGB images. These features feed into a shared dense layer containing 1536 nodes, which represents a holistic "understanding" of the plant's morphology.
  • Specialized Heads: From this shared layer, the network branches into three specialized "heads":
  • Leaf Count: A regression branch predicting the total number of leaves.
  • Projected Leaf Area (PLA): A regression branch predicting the percentage of the image covered by plant tissue.
  • Genotype Classification: A classification branch identifying whether the plant is a "wild-type" (WT) or a specific mutant.
Detailed architecture of the modified ResNet50 multitask network showing shared layers and task-specific branches.

By training the model end-to-end, the shared layers are forced to learn features that are useful for all tasks simultaneously. This "cross-talk" between tasks acts as a form of regularization, preventing the model from overfitting on any single trait and improving overall generalization.

Experimental Setup

The study utilized three publicly available datasets of Arabidopsis thaliana:

  • Ara2013: Contains 24 plants across 5 different genotypes, captured over 26 days. This served as the primary dataset for genotype classification.
  • A1 and A4: Datasets from the Computer Vision Problems in Plant Phenotyping (CVPPP) challenge, used to evaluate the model's ability to generalize across different experimental setups and image qualities.

Key Results and Insights

Synergy and Accuracy

The most significant finding is the synergy between tasks. Learning to estimate PLA and genotype significantly improves the model's ability to count leaves. The MTL approach reduced the Mean Squared Error (MSE) for leaf counting by more than 40% compared to a standard single-task network. The model achieved an $R^2$ value of 0.95 for leaf counting, indicating a very high correlation between predicted and actual counts.

Label Efficiency and Robustness

The framework demonstrates remarkable robustness to missing data. The authors showed that by leveraging area and genotype labels, researchers can reduce leaf count labels by 75% and still achieve a model that outperforms a single-task model trained on 100% of the labels. When leaf count labels were reduced to only 25%, the MTL model's performance remained stable (MSE ~1.13), while the single-task model's performance collapsed (MSE ~17.2).

Interpretability

Using "sliding window" occlusion tests, the researchers found that the MTL model focuses more accurately on actual plant tissue compared to single-task models, which were more easily distracted by background elements like soil or pot edges. Additionally, the study found that labeling mature plants is more effective than labeling juvenile ones, as mature plants provide the complex structural information necessary for the model to master the counting task.

Conclusion and Future Work

This multitask approach proves that we can "do more with less" in plant phenotyping. By training on related traits, we can build more accurate models with significantly less manual annotation effort. While highly effective for the rosette-style growth of Arabidopsis, future work should explore whether this multitask synergy holds for more complex 3D structures like maize or wheat, and how the framework performs under varying greenhouse or field conditions.

Summary updated

Abstract

Image-based plant phenotyping has been steadily growing and this has steeply increased the need for more efficient image analysis techniques capable of evaluating multiple plant traits. Deep learning has shown its potential in a multitude of visual tasks in plant phenotyping, such as segmentation and counting. Here, we show how different phenotyping traits can be extracted simultaneously from plant images, using multitask learning (MTL). MTL leverages information contained in the training images of related tasks to improve overall generalization and learns models with fewer labels. We present a multitask deep learning framework for plant phenotyping, able to infer three traits simultaneously: (i) leaf count, (ii) projected leaf area (PLA), and (iii) genotype classification. We adopted a modified pretrained ResNet50 as a feature extractor, trained end-to-end to predict multiple traits. We also leverage MTL to show that through learning from more easily obtainable annotations (such as PLA and genotype) we can predict a better leaf count (harder to obtain annotation). We evaluate our findings on several publicly available datasets of top-view images of <i>Arabidopsis thaliana</i>. Experimental results show that the proposed MTL method improves the leaf count mean squared error (MSE) by more than 40%, compared to a single task network on the same dataset. We also show that our MTL framework can be trained with up to 75% fewer leaf count annotations without significantly impacting performance, whereas a single task model shows a steady decline when fewer annotations are available. Code available at https://github.com/andobrescu/Multi_task_plant_phenotyping.
BibTeX
@inproceedings{dobrescu2020doing,
  author = {Andrei Dobrescu and Mario Valerio Giuffrida and Sotirios A. Tsaftaris},
  title = {{Doing More With Less: A Multitask Deep Learning Approach in Plant Phenotyping}},
  booktitle = {Frontiers in Plant Science},
  year = {2020},
  doi = {10.3389/fpls.2020.00141},
}
Figure