TL;DR
SegAnyFruit (SAF) is a new framework that distills the power of massive foundation models into lightweight, prompt-free segmenters optimized for agriculture. By leveraging a massive new auto-labeled web dataset (WebFruits) and a refined agricultural dataset (PaperFruits), SAF achieves state-of-the-art zero-shot performance—outperforming even SAM 3—while remaining fast enough to run in real-time on edge devices like the Jetson Orin NX.
What Problem Does This Paper Address?
Instance segmentation in agriculture is notoriously difficult. Fruits vary wildly in shape, color, and texture, and they often grow in dense, occluded clusters under unpredictable lighting.
Current solutions generally fall into two camps:
- Closed-set Specialists: Models like YOLO-seg are fast but brittle; they often fail when moved to a new orchard or presented with an unseen fruit variety.
- Foundation Models: Models like Segment Anything (SAM) generalize beautifully but are computationally heavy, require manual prompting (points or boxes) at inference, and often over-segment complex fruits (e.g., treating every individual drupelet of a blackberry as a separate object).
The authors aim to bridge this gap, creating a "foundation-level" fruit segmenter that is lightweight, prompt-free, and robust across diverse environments.
Methods
The core of the paper is the SAF Training Recipe, a three-stage progressive distillation process:
- Pre-training (Generic): Models are initialized on the massive SA-1B dataset to learn basic objectness and boundary priors.
- Mid-training (Broad): Models are trained on WebFruits, a new dataset of ~570K images. Labels are generated automatically using a VLM-guided pipeline. To ensure the model learns "whole fruit" concepts, the authors developed a Category-Aware Mask Curation pipeline. This uses geometric filters and VLM checks to merge over-segmented parts into single, coherent fruit instances.
- Post-training (Refined): Finally, the model is refined on PaperFruits, a collection of human-verified agricultural data, to master the nuances of dense orchard scenes and overlapping instances.
The authors also explored Generative Augmentation, using a video synthesis pipeline to create synthetic variations in lighting and ripeness, which proved particularly effective in boosting performance for rare or difficult-to-image conditions.
Datasets
The paper introduces a massive scale of curated resources:
- WebFruits: 571,494 images with 6.9M auto-generated masks across 127 fruit types.
- PaperFruits: 160,981 images with 2.7M masks consolidated from 36 agricultural sources.
- WebOOD: A dedicated zero-shot benchmark featuring 38 fruit types strictly excluded from training.
- Evaluation Suite: Seven manually annotated benchmarks (e.g., MSU-Apple, KFuji-Apple, StrawDI) used to test generalization across species and lighting.
Key Insights and Ideas
- Distillation over Prompting: The authors demonstrate that you don't need a giant model at runtime if you use it as a "teacher" to create high-quality training data for a smaller "student."
- The "Whole-Fruit" Prior: Standard foundation models often fail in agriculture because they lack a semantic understanding of what constitutes an "instance." The authors' curation pipeline (shown in Figure 2C) is critical for correcting SAM's tendency to fragment fruits.
- Resilience to Forgetting: One of the most striking findings is that SAF-initialized models are much more resistant to "catastrophic forgetting." When fine-tuned on a specific new domain, they retain their general fruit-segmentation abilities far better than models initialized from standard COCO or ImageNet weights.
- Cross-Domain Versatility: While designed for fruit, the "Pre-Mid-Post" recipe generalizes to other data-scarce domains like underwater salient objects, microscopy cells, and occluded humans, often matching the performance of models trained on 100x more manual labels.
Key Results
- Zero-Shot Superiority: The SAF-RF-DETR-XL model achieved an average mask AP50:95 of 69.06 across seven benchmarks, a +4.76 point improvement over the much larger SAM 3.
- Edge Performance: The SAF-YOLO26m-seg variant runs at 69 FPS on a Jetson Orin NX (FP16), making it truly field-ready for robotic harvesters.
- Label Efficiency: By using the Mid-training stage, the authors could recover 98.8% of full-data performance using 100x fewer manual labels.
- Qualitative Quality: SAF models produce significantly cleaner boundaries and fewer false positives in clustered environments compared to general-purpose VLMs.
Limitations and Future Work
Despite the impressive results, the authors acknowledge several areas for improvement:
- Pseudo-mask Errors: Automatically generated labels still contain occasional errors in boundary precision and instance granularity.
- Species Agnosticism: Currently, SAF treats "fruit" as a unified concept. It does not distinguish between, for example, an apple and a pear. Future work will look into fine-grained species recognition.
- Annotation Quality: There is a plan to further improve the automated annotation pipeline to achieve even sharper boundaries without human intervention.
Summary updated