GMT: Guided Mask Transformer for Leaf Instance Segmentation

Feng Chen, Sotirios A. Tsaftaris, Mario Valerio Giuffrida

Abstract

Leaf instance segmentation is a challenging multi-instance segmentation task, aiming to separate and delineate each leaf in an image of a plant. Accurate segmentation of each leaf is crucial for plant-related applications such as the fine-grained monitoring of plant growth and crop yield estimation. This task is challenging because of the high similarity (in shape and colour), great size variation, and heavy occlusions among leaf instances. Furthermore, the typically small size of annotated leaf datasets makes it more difficult to learn the distinctive features needed for precise segmentation. We hypothesise that the key to overcoming the these challenges lies in the specific spatial patterns of leaf distribution. In this paper, we propose the Guided Mask Transformer (GMT), which leverages and integrates leaf spatial distribution priors into a Transformer-based segmentor. These spatial priors are embedded in a set of guide functions that map leaves at different positions into a more separable embedding space. Our GMT consistently outperforms the state-of-the-art on three public plant datasets.

Introduction

In plant phenotyping, accurately counting and measuring individual leaves is vital for monitoring growth and predicting crop yields. However, leaf instance segmentation is notoriously difficult. Leaves often have nearly identical colors and shapes, vary wildly in size, and frequently overlap.

While Transformer-based models like Mask2Former have set new benchmarks in general computer vision, they often struggle with plant imagery. Common failure modes include missing small leaves at the plant's center, merging overlapping leaves into a single blob, or confusing green background objects with foliage. This is exacerbated by the fact that plant datasets are typically small (hundreds of images), making it hard for large models to learn distinctive features from scratch.

Methods: Integrating Spatial Priors

The Guided Mask Transformer (GMT) extends the Mask2Former architecture by baking "spatial priors"—knowledge of how leaves naturally grow and cluster—directly into the model. GMT introduces three novel components designed to leverage the predictable spatial distribution of leaves (e.g., growing outward from a central point).

The GMT architecture integrates spatial priors through Guided Positional Encoding (GPE), Guided Dynamic Positional Queries (GDPQ), and a Guided Embedding Fusion Module (GEFM).
  1. Guide Functions: The model uses harmonic (sinusoidal) functions to map pixel coordinates into an embedding space. These functions are trained to ensure that pixels belonging to different leaf instances are pushed far apart, making them easier to distinguish.
  2. Guided Positional Encoding (GPE): Standard Transformers use positional encodings to represent spatial locations. GMT enhances this by adding learned spatial priors from the guide functions, helping the model align its spatial "understanding" with actual plant structures.
  3. Guided Dynamic Positional Queries (GDPQ): Instead of starting with static queries, GMT dynamically generates queries based on intermediate mask predictions. These queries are "conditioned" by the guide functions, essentially guiding the model on where to look for the next leaf based on previous detections.
  4. Guided Embedding Fusion Module (GEFM): This module fuses standard image features with specialized "guided" embeddings. An auxiliary $L_1$ loss ensures the model learns to separate instances effectively in this new space.

Key Results

GMT was evaluated on three public datasets: CVPPP LSC (rosette plants), MSU-PID (Arabidopsis and bean plants), and KOMATSUNA (Japanese mustard spinach). The model consistently outperformed the baseline Mask2Former and other state-of-the-art (SOTA) methods.

Qualitative comparison showing GMT's superior performance in segmenting small and overlapping leaves compared to Mask2Former across various datasets.
  • Accuracy (SBD): On the CVPPP LSC dataset, GMT achieved a Symmetric Best Dice (SBD) of 90.1, compared to Mask2Former’s 89.5. On MSU-PID and KOMATSUNA, it reached SBD scores of 83.5 and 91.0 respectively.
  • Counting Precision (|DiC|): GMT significantly reduced the "Difference in Count" error. In CVPPP LSC, the error dropped from 0.67 to 0.48, indicating much better leaf counting performance.
  • Small Leaf Performance: Qualitative results show GMT is significantly more robust at identifying tiny, young leaves at the center of the plant—areas where other models typically ignore or merge instances.

Discussion and Future Work

The success of GMT stems from exploiting biological regularity. By encoding growth patterns into the architecture, the model requires less data to achieve high precision. However, the authors noted that increasing the dimensionality of guide functions is computationally expensive. While they used a "post-processing" expansion trick to mitigate this, the model still adds overhead compared to a vanilla Transformer.

Future work includes extending GMT to more complex agricultural scenarios, such as vine crops or dense forest canopies where structures are less centralized. Additionally, optimizing the architecture for deployment on edge devices like agricultural drones or robots remains a key goal.

Summary updated

Abstract

Leaf instance segmentation is a challenging multi-instance segmentation task, aiming to separate and delineate each leaf in an image of a plant. Accurate segmentation of each leaf is crucial for plant-related applications such as the fine-grained monitoring of plant growth and crop yield estimation. This task is challenging because of the high similarity (in shape and colour), great size variation, and heavy occlusions among leaf instances. Furthermore, the typically small size of annotated leaf datasets makes it more difficult to learn the distinctive features needed for precise segmentation. We hypothesise that the key to overcoming the these challenges lies in the specific spatial patterns of leaf distribution. In this paper, we propose the Guided Mask Transformer (GMT), which leverages and integrates leaf spatial distribution priors into a Transformer-based segmentor. These spatial priors are embedded in a set of guide functions that map leaves at different positions into a more separable embedding space. Our GMT consistently outperforms the state-of-the-art on three public plant datasets. Our code is available at https://github.com/vios-s/gmt-leaf-ins-seg.
BibTeX
@inproceedings{chen2025gmt,
  author = {Feng Chen and Sotirios A. Tsaftaris and Mario Valerio Giuffrida},
  title = {{GMT: Guided Mask Transformer for Leaf Instance Segmentation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  url = {https://openreview.net/forum?id=3efPjV79nl},
}
Figure