Spatio-Temporal Metric-Semantic Mapping for Persistent Orchard Monitoring: Method and Dataset

Xu Liu, Jiuzhou Lei, Ankit Prabhu, Xu Liu 0007, Fernando Cladera, Mehrad Mortazavi, Reza Ehsani, Pratik Chaudhari, Vijay Kumar

TL;DR

This paper introduces a novel 4D spatio-temporal metric-semantic mapping system for orchards, enabling precise tracking of individual fruit growth over an entire season. By fusing LiDAR and RGB data, their method accurately counts fruits, estimates their sizes, and associates them across time, significantly improving upon existing techniques and releasing a comprehensive multimodal dataset for future research.

What Problem Does This Paper Address?

Monitoring orchards at the individual fruit level throughout the growing season is crucial for optimizing agricultural practices like chemical thinning, irrigation, and yield prediction. While existing methods can map orchards in 3D or track plant growth, they often lack the fine-grained detail needed to monitor individual fruits across multiple time sessions. This gap makes it difficult to understand fruit development, estimate precise yields, and implement targeted interventions. The challenge lies in accurately associating fruits across different time points, despite their positional changes, variations in appearance, and potential occlusions.

Methods

The proposed system tackles this challenge through a multi-stage approach:

  • LiDAR-RGB Fusion for 3D Fruit Localization: The system combines LiDAR point clouds for depth estimation with RGB images for fruit instance segmentation (using a fine-tuned YOLOv8 model). This fusion allows for the extraction of 3D point clouds for each detected fruit. To improve robustness, point clouds from a short time window (1.5 seconds) are accumulated.
  • Tracking Across Image Frames: Fruits are tracked across consecutive image frames using a Hungarian assignment algorithm. The cost function for matching fruit masks is based on the Intersection over Union (IoU) of their projected point clouds. A cost matrix incorporating both matched masks and unmatched masks (to handle false negatives) is used. Fruits observed in at least three consecutive frames are considered reliable tracks to avoid overcounting.
  • Re-projection Error Minimization: To refine fruit positions and camera poses, a nonlinear least squares problem is solved to minimize the reprojection error across all image frames. This step accounts for noise in point clouds and variations in fruit observation from different viewpoints.
  • Spatio-temporal (4D) Tracking: This is the core of the temporal association. It's a two-stage matching process:
  • Initial Matching: For each image in session A, the most similar image in session B is identified. Fruits are then matched using a cost function that combines visual similarity (using DinoV2 features) and positional (Euclidean) distance.
  • Topology-aware Matching: A second stage refines matches by incorporating a "topology cost." This cost leverages the spatial relationships between fruits within an image, represented by vectors pointing from a target fruit to its neighbors. This helps reject false matches that violate consistent spatial arrangements.
  • Metric-Semantic Map Generation: The output is a 4D map that provides actionable information like fruit counts, sizes, and positions over time.

Datasets

The paper introduces a new multimodal dataset collected from real-world orchards over an entire growth season (April to August). This dataset includes:

  • LiDAR point clouds
  • High-resolution RGB images
  • GPS and IMU measurements

It covers five different fruit species (apple, pear, peach, cherry, and pistachio) across 284 trees. This dataset is notable for its extended temporal coverage compared to existing datasets, which often capture only a single growth stage.

Key Insights and Ideas

  • The Power of 4D: Tracking fruits not just in 3D space but also over time (4D) is essential for understanding growth dynamics and enabling precise agricultural management.
  • LiDAR-RGB Synergy: Combining the geometric accuracy of LiDAR with the semantic richness of RGB images is a powerful strategy for robust fruit detection and localization.
  • Topology as a Constraint: Incorporating the spatial relationships between fruits (topology) significantly improves the accuracy of temporal association, helping to reject spurious matches that might arise from visual or positional similarities alone.
  • Dataset for Temporal Analysis: The release of a dataset spanning an entire growth cycle is a significant contribution, enabling research into long-term fruit development and phenotyping.

Key Results

  • 3D Fruit Counting: Achieved a 3.1% total count error on 1790 apples across 60 trees, with a mean absolute count error of 6 apples per batch.
  • Fruit Size Estimation: Demonstrated a mean absolute error of 1.10 cm for fruit size estimation.
  • 4D Fruit Association: Improved precision by 23.7% and F1 score by 11.9% compared to a strong baseline, particularly in challenging scenarios with occlusions and missed detections. The method showed robustness even when up to 50% of data was removed.

Limitations and Future Work

  • LiDAR Sparsity: The paper acknowledges that LiDAR sparsity can limit the localization and size estimation accuracy for small fruits.
  • Topology Cost Sensitivity: While effective, the topology cost can sometimes incorrectly reject valid matches.
  • Autonomous Robotics: The authors express a future goal of enabling autonomous robots (like UAVs) for data collection and orchard monitoring, suggesting the current hand-carried sensor setup is a stepping stone.

Summary updated

Abstract

Monitoring orchards at the individual tree or fruit level throughout the growth season is crucial for plant phenotyping and horticultural resource optimization, such as chemical use and yield estimation. We present a 4D spatio-temporal metric-semantic mapping system that integrates multi-session measurements to track fruit growth over time. Our approach combines a LiDAR-RGB fusion module for 3D fruit localization with a 4D fruit association method leveraging positional, visual, and topology information for improved data association precision. Evaluated on real orchard data, our method achieves a 96.9% fruit counting accuracy for 1,790 apples across 60 trees, a mean fruit size estimation error of 1.1 cm, and a 23.7% improvement in 4D data association precision over baselines. We publicly release a multimodal dataset covering five fruit species across their growth seasons at https://4d-metric-semantic-mapping.org/
BibTeX
@article{liu2025spatio,
  author = {Xu Liu and Jiuzhou Lei and Ankit Prabhu and Xu Liu and Fernando Cladera and Mehrad Mortazavi and Reza Ehsani and Pratik Chaudhari and Vijay Kumar},
  title = {{Spatio-Temporal Metric-Semantic Mapping for Persistent Orchard Monitoring: Method and Dataset}},
  journal = {IEEE RA-L},
  year = {2025},
  doi = {10.1109/lra.2025.3588037},
}
Figure