Multimodal Data Fusion for Power-On-and-GoRobotic Systems in Retail

Shubham D. Sonawani, Kailas Maneparambil, Heni Ben AmorarXiv 2021 · OpenReview

TL;DR

This paper presents a "power-on-and-go" (PO&G) robotic system for retail environments that leverages multimodal data fusion from low-cost sensors. By combining a monocular camera with Bluetooth Low Energy (BLE) beacons, the system generates high-quality 3D environmental maps for navigation and manipulation at a fraction of the cost of traditional solutions.

What Problem Does This Paper Address?

The retail and warehouse industries are ripe for automation, with tasks like shelf replenishment and product retrieval being prime candidates for robotic solutions. However, the widespread adoption of robots in these environments is hindered by two major challenges: high acquisition costs and the constant need for calibration and human intervention. This means robots aren't "plug-and-play" ready. The authors aim to overcome these barriers by developing an affordable, self-initializing robotic system that can generate rich environmental data for navigation and manipulation without extensive setup.

Methods

The core of the proposed system is a multimodal data fusion approach. It combines:

  • Active Sensing (Monocular Camera): A standard monocular camera is used with a deep learning model (CNN) to perform per-pixel depth estimation. This allows the generation of accurate 3D point clouds of the environment. The network is trained using a loss function that considers pixel-wise depth, gradient, and structural similarity.
  • Passive Sensing (BLE Beacons): A network of low-cost Bluetooth Low Energy (BLE) beacons is deployed in the workspace. The robot uses the received signal strength indicator (RSSI) from these beacons to estimate its relative position and orientation.
  • Inertial Measurement Unit (IMU) and Wheel Encoders: The robot's internal IMU and wheel encoders provide odometry information, which is also integrated into the localization process.

These data streams are fused using an Extended Kalman Filter (EKF) to estimate the robot's position and orientation. This estimated pose is then used to align and refine the individual 3D point clouds generated by the camera, ultimately building a comprehensive map of the environment. The system aims for "power-on-and-go" functionality, meaning minimal setup and calibration are required.

Datasets

Not explicitly discussed in the paper. The authors mention using "ground truth depth maps" for training their depth estimation network, implying the use of existing datasets for this purpose.

Key Insights and Ideas

  • Artificial Sensors for Cost Reduction: The paper champions the idea of "artificial sensors" – ML models that synthesize high-quality data from low-cost sensors. This bypasses the need for expensive, specialized hardware like LIDAR.
  • Symbiotic Integration: The key insight is that a symbiotic integration of physical sensors (camera, BLE, IMU) and ML-based inference is crucial for achieving both cost-efficiency and high sensing capabilities.
  • BLE for Robust Localization: BLE beacons, despite their low cost and lack of calibration requirements, prove effective for robust localization, especially when combined with other sensor data. This is a significant advantage for PO&G systems.
  • Overcoming Sensor Limitations: The trained depth estimation network shows an ability to interpolate data, potentially overcoming limitations of traditional sensors like missing values or limited range, and is well-suited for grasping tasks.

Key Results

  • Depth Estimation Accuracy: The depth prediction network achieves an approximate accuracy of 91% on test data.
  • Point Cloud Generation Rate: Inference for the depth model and point cloud generation is performed at a rate of 9Hz.
  • BLE Data Update Rate: BLE sensor data is obtained and updated at a rate of 10Hz.
  • Qualitative Results: The system successfully generates 3D point clouds and approximates the environment, even showing promise in accurately representing corridors. The approach is shown to work for objects very close to the camera (< 20cm), which is beneficial for manipulation.

Limitations and Future Work

  • Multimodal Integration: Currently, the multimodal integration is largely performed through hand-coded operations.
  • Future Work: The authors envision learning the data fusion step itself. This would allow the system to discover more complex methods for cross-referencing information across modalities, leading to a richer, single model of the world. They also aim to enable the robot to navigate and grasp/manipulate objects.

Summary updated

Abstract

Robotic systems for retail have gained a lot of attention due to the labor-intensive nature of such business environments. Many tasks have the potential to be automated via intelligent robotic systems that have manipulation capabilities. For example, empty shelves can be replenished, stray products can be picked up or new items can be delivered. However, many challenges make the realization of this vision a challenge. In particular, robots are still too expensive and do not work out of the box. In this paper, we discuss a work-in-progress approach for enabling power-on-and-go robots in retail environments through a combination of active, physical sensors and passive, artificial sensors. In particular, we use low-cost hardware sensors in conjunction with machine learning techniques in order to generate high-quality environmental information. More specifically, we present a setup in which a standard monocular camera and Bluetooth low-energy yield a reliable robot system that can immediately be used after placing a couple of sensors in the environment. The camera information is used to synthesize accurate 3D point clouds, whereas the BLE data is used to integrate the data into a complex map of the environment. The combination of active and passive sensing enables high-quality sensing capabilities at a fraction of the costs traditionally associated with such tasks.
BibTeX
@misc{sonawani2021multimodal,
  author = {Shubham D. Sonawani and Kailas Maneparambil and Heni Ben Amor},
  title = {{Multimodal Data Fusion for Power-On-and-GoRobotic Systems in Retail}},
  year = {2021},
  eprint = {2103.12241},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.12241},
}
Figure