Push Past Green: Learning to Look Behind Plant Foliage by Moving It

Xiaoyu Zhang, Saurabh Gupta

Abstract

Autonomous agriculture applications, such as inspection, phenotyping, and fruit harvesting, require manipulating plant foliage to reveal what lies behind leaves and branches. However, partial visibility, extreme clutter, thin structures, and the unknown geometry and dynamics of plants make such manipulation exceptionally challenging. We tackle these challenges through a data-driven framework called PushPastGreen (PPG). We utilize self-supervision to train SRPNet, a neural network that predicts the specific space revealed upon the execution of a candidate action. By integrating SRPNet with the Cross-Entropy Method (CEM), we can predict and execute actions that effectively reveal occluded areas. Because SRPNet predicts not just the quantity but the spatial distribution of revealed space, it enables sequences of actions that incrementally uncover the environment. We evaluate PPG on synthetic vines and real Dracaena plants across five settings, including tests for generalization to novel configurations and plant growth. Our results demonstrate that PPG significantly outperforms hand-crafted exploration methods and traditional dynamics models.

Introduction

In the pursuit of fully autonomous agriculture, robots must navigate environments that are inherently "messy." Unlike the structured settings of a factory floor, a farm or greenhouse is filled with deformable, overlapping, and complex biological structures. One of the primary hurdles for robotic systems in these environments is self-occlusion: the tendency of a plant's own foliage to hide targets of interest, such as ripening fruit or signs of disease.

Traditional robotic approaches often treat plants as rigid obstacles to be avoided. However, to truly interact with a crop, a robot must treat foliage as a deformable barrier that can be intelligently moved. The "Push Past Green" (PPG) framework addresses this by teaching robots how to peek behind the leaves through active manipulation.

The Challenge of Plant Dynamics

Plants are a "nightmare" for classical robotics modeling for several reasons:

  • Complex Geometry: Thin stems and intricate leaf patterns are difficult for standard depth sensors to resolve.
  • Unknown Dynamics: Every plant is unique. Leaves bend, twist, and spring back in ways that are difficult to simulate accurately without high-fidelity, computationally expensive physics models.
  • Clutter: In dense foliage, moving one leaf often moves several others, creating a chain reaction of visibility changes.

To bypass the "sim-to-real" gap—where models trained in simulation fail in the real world—PPG relies on self-supervised learning directly on physical hardware.

The PPG Framework

The PPG system consists of two core components designed to turn raw visual data into effective manipulation strategies:

1. SRPNet (Space-Revealed Prediction Network)

Instead of attempting to model the full physics of the plant, the Space-Revealed Prediction Network (SRPNet) focuses solely on the outcome of an action. Given an RGB-D image of the plant and a proposed "push" action (defined by direction, height, and distance), SRPNet predicts a spatial probability map. This map indicates exactly where new space is likely to be revealed. By using a UNet-style architecture, the network can process local image patches to understand the orientation of leaves and predict how they will displace.

2. Control via CEM

To decide on the optimal move, the system employs the Cross-Entropy Method (CEM). The robot samples hundreds of potential push actions and evaluates each one using SRPNet. It then selects the action predicted to reveal the most previously unseen space. Crucially, because SRPNet provides a spatial map, the system can maintain a "memory" of what has already been explored, allowing it to plan multi-step sequences that target different occluded zones in succession.

Experimental Setup and Data Collection

The researchers trained the system using a Franka Emika robot arm equipped with an Azure Kinect depth camera. The training was entirely self-supervised: the robot interacted with plants for dozens of hours, recording the state before and after each push to learn the relationship between its movements and the resulting visibility.

Data was collected across two primary scenarios:

  • Decorative Vines: 3,529 interactions representing a 2D exploration challenge with highly entangled structures.
  • Dracaena Plant: 2,175 interactions representing a 3D challenge with large, heavy leaves that exhibit specific bending behaviors.

To test the robustness of the model, the team also evaluated generalization on "Sparse" and "Separated" vine configurations, as well as a Dracaena plant after two months of additional growth. This "June vs. August" test was designed to see if the model could handle changes in leaf length and foliage density over time.

Key Insights and Results

Spatial Awareness Matters

A key finding of the research is that predicting where space is revealed is far more effective than simply predicting how much space is revealed. This spatial awareness allows PPG to avoid redundant actions and systematically clear a field of view.

Quantitative Performance

In long-horizon exploration tasks (10-step sequences), PPG demonstrated a relative improvement in revealed space of 4% to 34% over competitive hand-crafted baselines. In single-action selection, the model showed a 62% improvement for vines and a 49% improvement for the Dracaena plant compared to random horizontal pushes. This indicates that the model successfully learned to identify high-leverage interaction points rather than simply pushing at random.

Generalization to Growth

One of the most promising results was the model's ability to generalize to plant growth. A model trained on the Dracaena in June remained effective in August, despite the plant's significantly altered appearance. This suggests that the network learned fundamental principles of leaf manipulation that transcend specific leaf sizes.

Limitations and Future Work

While PPG represents a significant step forward, several challenges remain:

  • The "Optimism" Trap: The model occasionally becomes overly optimistic, repeatedly attempting a failed action because it predicts a high reward that never materializes.
  • Open-Loop Execution: The current system is open-loop; it pushes and then looks. Future iterations could benefit from closed-loop visual or tactile feedback to adjust the push in real-time if a leaf slips.
  • Plant Safety: The current model does not explicitly account for the structural integrity of the plant. Incorporating force sensors could help the robot ensure it doesn't snap stems or tear leaves during exploration.

Conclusion

Push Past Green demonstrates that self-supervised, data-driven methods can overcome the complexities of plant manipulation. By focusing on predicting revealed space rather than complex physics, PPG provides a robust framework for robotic vision in dense, occluded agricultural environments.

Summary updated

Abstract

Autonomous agriculture applications (e.g., inspection, phenotyping, plucking fruits) require manipulating the plant foliage to look behind the leaves and the branches. Partial visibility, extreme clutter, thin structures, and unknown geometry and dynamics for plants make such manipulation challenging. We tackle these challenges through data-driven methods. We use self-supervision to train SRPNet, a neural network that predicts what space is revealed on execution of a candidate action on a given plant. We use SRPNet with the cross-entropy method to predict actions that are effective at revealing space beneath plant foliage. Furthermore, as SRPNet does not just predict how much space is revealed but also where it is revealed, we can execute a sequence of actions that incrementally reveal more and more space beneath the plant foliage. We experiment with a synthetic (vines) and a real plant (Dracaena) on a physical test-bed across 5 settings including 2 settings that test generalization to novel plant configurations. Our experiments reveal the effectiveness of our overall method, PPG, over a competitive hand-crafted exploration method, and the effectiveness of SRPNet over a hand-crafted dynamics model and relevant ablations.
BibTeX
@inproceedings{zhang2023push,
  author = {Xiaoyu Zhang and Saurabh Gupta},
  title = {{Push Past Green: Learning to Look Behind Plant Foliage by Moving It}},
  booktitle = {Conference on Robot Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=yDu5GwpWaK},
}
Figure