Mushroom Detection and Three Dimensional Pose Estimation from Multi-View Point Clouds

George Retsinas, Niki Efthymiou, Dafni Anagnostopoulou, Petros Maragos

TL;DR

This paper introduces a novel, annotation-free pipeline for detecting and estimating the 3D pose of mushrooms using multi-view point clouds from RGB-D cameras. This is crucial for developing robotic harvesters that can precisely grasp mushrooms, addressing the significant challenge of limited 3D annotated data in agriculture.

What Problem Does This Paper Address?

The agricultural robotics field is rapidly advancing, with a growing need for sophisticated perception systems to enable robots to perform delicate tasks like harvesting. Specifically, mushroom harvesting requires robots to accurately identify individual mushrooms and determine their precise 3D position and orientation (pose) to enable successful grasping and removal. However, a major bottleneck in developing such systems is the scarcity of 3D annotated data, which is extremely time-consuming and difficult to acquire at scale. Existing methods often rely on 2D detection or struggle with the inherent symmetry and clustering of mushrooms. This paper tackles the problem of robust mushroom detection and 3D pose estimation in cluttered environments, specifically for industrial mushroom farms, without requiring extensive 3D annotations.

Methods

The core of the proposed solution is a two-stage, annotation-free pipeline:

  1. Mushroom Instance Segmentation: This stage aims to isolate individual mushrooms from the background and from each other. It leverages a one-class classification approach using k-Medoids clustering on FPFH (Fast Point Feature Histograms) features extracted from a single 3D mushroom template. The key innovation here is the use of template augmentation – creating numerous variations of the 3D mushroom template through transformations and deformations. This augmented template set allows the k-Medoids model to learn robust, mushroom-specific features, enabling accurate segmentation even when mushrooms are close together. A subsequent density clustering step (DBSCAN) is then applied to achieve instance segmentation, separating individual mushrooms.
  2. Template Iterative Fine-Tuning (3D Pose Estimation): Once individual mushrooms are segmented, their 3D pose is estimated by aligning a 3D mushroom template to the segmented point cloud. This is achieved using a modified Iterative Closest Point (ICP) algorithm. The modification involves using k-nearest neighbors and selecting the neighbor with the highest feature similarity (cosine similarity) to guide the alignment, allowing for larger steps and more robust convergence. An ellipsoid fitting alternative is also explored as a faster, though less accurate, method due to the mushroom cap's shape.

The pipeline is designed to work with multi-view point clouds, which are preprocessed through plane segmentation and alignment to create a unified scene representation.

Datasets

  • Synthetic Dataset: The authors generated a synthetic dataset of mushroom scenes to enable quantitative evaluation. This dataset was created using a custom scene generator that places mushrooms on a realistic ground plane, applying various transformations (translation, scaling, rotation, local deformations) and simulating realistic point cloud acquisition with hidden point removal. This approach circumvents the need for real-world 3D annotations.
  • Real Data: Qualitative validation was performed on real data collected using multiple RealSense RGB-D cameras in different configurations, as described in Section 3 of the paper.

Key Insights and Ideas

  • Annotation-Free Learning is Key: The most significant insight is the successful development of a robust pipeline that bypasses the need for large-scale 3D annotations, a major hurdle in agricultural robotics. This is achieved by leveraging a single 3D template model.
  • Template Augmentation for Robust Segmentation: Simply using a single 3D template for feature extraction proved insufficient. Augmenting the template with various transformations and deformations was crucial for learning discriminative features that generalize well to different mushroom instances and orientations.
  • k-Medoids for Feature Clustering: Using k-Medoids for clustering the augmented template features proved effective for one-class classification, allowing for intuitive distance-based segmentation.
  • Modified ICP for Pose Estimation: The standard ICP algorithm can be sensitive to initialization and object symmetry. The proposed modification, incorporating feature similarity for neighbor selection, significantly improves its robustness for mushroom pose estimation.
  • Multi-View Fusion is Beneficial: While not the sole focus, the use of multi-view point clouds aids in creating a more complete 3D representation, reducing occlusion issues and improving overall scene understanding.

Key Results

  • High Detection Accuracy: The proposed pipeline achieved a Mean Average Precision (MAP) of 99.80% at a 25% IoU threshold and 96.31% at a 50% IoU threshold on their synthetic dataset, significantly outperforming a standard RANSAC-based approach (53.62% at 50% IoU).
  • Accurate Pose Estimation: Cosine similarity between ground truth and estimated rotation vectors was consistently high (e.g., 0.9920 at 50% IoU), with a mean angle error of 7.27°.
  • Robustness to Clutter: The method demonstrated effectiveness even when mushrooms were very close to each other, a common scenario in mushroom cultivation.
  • Efficiency: The RANSAC-based template matching achieved near real-time performance (~0.2s), while the proposed pipeline's processing time was acceptable for the task, with potential for further optimization.

Limitations and Future Work

  • Computational Overhead: While efficient, the proposed pipeline still involves multiple steps. The optional mesh reconstruction step was found to be redundant and computationally expensive.
  • Ellipsoid Alternative Limitations: The ellipsoid fitting alternative, while faster, suffered from ambiguity due to the mushroom's symmetry and partial point cloud information, leading to erroneous pose estimations in some cases.
  • Future Work: The authors suggest fine-tuning deep learning models on synthetic mushroom scenes for potentially even better performance. They also plan to explore constraint-based alternatives for ellipsoid fitting and further investigate more efficient ways to represent surface information directly on point clouds. The availability of real data is limited to the end of the SoftGrip project.

Summary updated

Abstract

Agricultural robotics is an up and coming field which deals with the development of robotic systems able to tackle a multitude of agricultural tasks efficiently. The case of interest, in this work, is mushroom collection in industrial mushroom farms. Developing such a robot, able to select and out-root a mushroom, requires delicate actions that can only be conducted if a well-performing perception module exists. Specifically, one should accurately detect the 3D pose of a mushroom in order to facilitate the smooth operation of the robotic system. In this work, we develop a vision module for 3D pose estimation of mushrooms from multi-view point clouds using multiple RealSense active–stereo cameras. The main challenge is the lack of annotation data, since 3D annotation is practically infeasible on a large scale. To address this, we developed a novel pipeline for mushroom instance segmentation and template matching, where a 3D model of a mushroom is the only data available. We evaluated, quantitatively, our approach over a synthetic dataset of mushroom scenes, and we, further, validated, qualitatively, the effectiveness of our method over a set of real data, collected by different vision settings.
BibTeX
@inproceedings{retsinas2023mushroom,
  author = {George Retsinas and Niki Efthymiou and Dafni Anagnostopoulou and Petros Maragos},
  title = {{Mushroom Detection and Three Dimensional Pose Estimation from Multi-View Point Clouds}},
  booktitle = {Sensors},
  year = {2023},
  url = {https://openreview.net/forum?id=CSHCDnzvOV},
}
Figure