SegAnyFruit: Learning Generalizable and Edge Deployable Fruit Instance Segmentation at Scale

Yanan Wang, Yuzheng wang, Yibin Ying, Zhenghao FeiarXiv 2026 · arXiv

TL;DR

SegAnyFruit (SAF) is a new framework that distills the power of massive foundation models into lightweight, prompt-free segmenters optimized for agriculture. By leveraging a massive new auto-labeled web dataset (WebFruits) and a refined agricultural dataset (PaperFruits), SAF achieves state-of-the-art zero-shot performance—outperforming even SAM 3—while remaining fast enough to run in real-time on edge devices like the Jetson Orin NX.

What Problem Does This Paper Address?

Instance segmentation in agriculture is notoriously difficult. Fruits vary wildly in shape, color, and texture, and they often grow in dense, occluded clusters under unpredictable lighting.

Current solutions generally fall into two camps:

  1. Closed-set Specialists: Models like YOLO-seg are fast but brittle; they often fail when moved to a new orchard or presented with an unseen fruit variety.
  2. Foundation Models: Models like Segment Anything (SAM) generalize beautifully but are computationally heavy, require manual prompting (points or boxes) at inference, and often over-segment complex fruits (e.g., treating every individual drupelet of a blackberry as a separate object).

The authors aim to bridge this gap, creating a "foundation-level" fruit segmenter that is lightweight, prompt-free, and robust across diverse environments.

Methods

The core of the paper is the SAF Training Recipe, a three-stage progressive distillation process:

  1. Pre-training (Generic): Models are initialized on the massive SA-1B dataset to learn basic objectness and boundary priors.
  2. Mid-training (Broad): Models are trained on WebFruits, a new dataset of ~570K images. Labels are generated automatically using a VLM-guided pipeline. To ensure the model learns "whole fruit" concepts, the authors developed a Category-Aware Mask Curation pipeline. This uses geometric filters and VLM checks to merge over-segmented parts into single, coherent fruit instances.
  3. Post-training (Refined): Finally, the model is refined on PaperFruits, a collection of human-verified agricultural data, to master the nuances of dense orchard scenes and overlapping instances.
Figure 1. Overview of SegAnyFruit. (A) Three-stage Pre–Mid–Post training recipe. (B) Prompt-free fruit segmentation across diverse scenes and unseen types.. (C) Efficient automatic annotation. (D) Cross-domain recipe validation. (E) Scale of our curated resources.

The authors also explored Generative Augmentation, using a video synthesis pipeline to create synthetic variations in lighting and ripeness, which proved particularly effective in boosting performance for rare or difficult-to-image conditions.

Datasets

The paper introduces a massive scale of curated resources:

  • WebFruits: 571,494 images with 6.9M auto-generated masks across 127 fruit types.
  • PaperFruits: 160,981 images with 2.7M masks consolidated from 36 agricultural sources.
  • WebOOD: A dedicated zero-shot benchmark featuring 38 fruit types strictly excluded from training.
  • Evaluation Suite: Seven manually annotated benchmarks (e.g., MSU-Apple, KFuji-Apple, StrawDI) used to test generalization across species and lighting.
Figure 2. WebFruits and PaperFruits. (A) Coverage and distributions. (B) Image and mask counts; Train includes the internal validation split, and PaperFruits Test summarizes the five held-out agricultural benchmarks. Collection labels include varieties and visual forms. (C) Whole-fruit mask curation. (D) Scene diversity.

Key Insights and Ideas

  • Distillation over Prompting: The authors demonstrate that you don't need a giant model at runtime if you use it as a "teacher" to create high-quality training data for a smaller "student."
  • The "Whole-Fruit" Prior: Standard foundation models often fail in agriculture because they lack a semantic understanding of what constitutes an "instance." The authors' curation pipeline (shown in Figure 2C) is critical for correcting SAM's tendency to fragment fruits.
  • Resilience to Forgetting: One of the most striking findings is that SAF-initialized models are much more resistant to "catastrophic forgetting." When fine-tuned on a specific new domain, they retain their general fruit-segmentation abilities far better than models initialized from standard COCO or ImageNet weights.
  • Cross-Domain Versatility: While designed for fruit, the "Pre-Mid-Post" recipe generalizes to other data-scarce domains like underwater salient objects, microscopy cells, and occluded humans, often matching the performance of models trained on 100x more manual labels.

Key Results

  • Zero-Shot Superiority: The SAF-RF-DETR-XL model achieved an average mask AP50:95 of 69.06 across seven benchmarks, a +4.76 point improvement over the much larger SAM 3.
  • Edge Performance: The SAF-YOLO26m-seg variant runs at 69 FPS on a Jetson Orin NX (FP16), making it truly field-ready for robotic harvesters.
  • Label Efficiency: By using the Mid-training stage, the authors could recover 98.8% of full-data performance using 100x fewer manual labels.
  • Qualitative Quality: SAF models produce significantly cleaner boundaries and fewer false positives in clustered environments compared to general-purpose VLMs.
Qualitative comparison showing SAF's superior boundary precision.

Limitations and Future Work

Despite the impressive results, the authors acknowledge several areas for improvement:

  • Pseudo-mask Errors: Automatically generated labels still contain occasional errors in boundary precision and instance granularity.
  • Species Agnosticism: Currently, SAF treats "fruit" as a unified concept. It does not distinguish between, for example, an apple and a pear. Future work will look into fine-grained species recognition.
  • Annotation Quality: There is a plan to further improve the automated annotation pipeline to achieve even sharper boundaries without human intervention.

Summary updated

Abstract

Deploying fruit instance segmentation in real agricultural settings requires generalizing across unseen species, orchards, and imaging conditions under strict onboard compute and annotation budgets. We introduce SegAnyFruit (SAF), a data construction and progressive training framework that distills foundation-model priors into lightweight, prompt-free segmenters. SAF combines generic mask pre-training, broad mid-training on automatically annotated web imagery from our curated WebFruits dataset, and human-verified agricultural refinement post-training on PaperFruits, together spanning 733K images and 9.66M whole-fruit masks across 165 fruit types. Across seven benchmarks, zero-shot SAF-RF-DETR-XL reaches 69.06 average mask AP50:95 (+4.76 over SAM 3) while remaining deployable on edge devices. SAF initialization also boosts few-shot to full-shot adaptation across five agricultural domains while mitigating catastrophic forgetting. Finally, we systematically analyze training stages, data scale, model capacity, and generative augmentation, and validate our pipeline's transferability to underwater, cell, and human segmentation. We will release our datasets, code, and SAF models.
BibTeX
@misc{wang2026seganyfruit,
  author = {Yanan Wang and Yuzheng wang and Yibin Ying and Zhenghao Fei},
  title = {{SegAnyFruit: Learning Generalizable and Edge Deployable Fruit Instance Segmentation at Scale}},
  year = {2026},
  eprint = {2611.25475},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25475},
}
Figure