Autonomous Wall Building with a UGV- UAV Team at MBZIRC 2020

Christian Lenz, Max Schwarz, Andre Rochow, Jan Razlaw, Arul Selvam Periyasamy, Michael Schreiber, Sven Behnke

TL;DR

This paper presents a coordinated robotic team—consisting of an omnidirectional ground vehicle (UGV) and a heavy-lift drone (UAV)—designed to autonomously build walls in a challenging outdoor environment. By combining high-precision LiDAR-based perception for the ground robot with real-time visual servoing for the aerial team, the researchers secured a second-place finish at the prestigious MBZIRC 2020 robotics competition.

What Problem Does This Paper Address?

Autonomous construction is a "holy grail" for robotics, with applications ranging from disaster response to building structures on other planets. However, moving from a controlled lab to an outdoor construction site introduces massive hurdles:

  1. Perception: Robots must identify and precisely locate bricks under harsh, changing sunlight without external motion-capture systems.
  2. Manipulation: Bricks are heavy (up to 2kg) and long (up to 1.8m), requiring grippers that are both strong and capable of handling misalignments.
  3. Logistics: Building a wall isn't just about picking up a brick; it’s about optimizing the sequence of movements to minimize battery drain and time.

The authors tackle these issues within the context of the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2020, where robots had to build complex walls from bricks of varying sizes and colors autonomously.

Methods

The team developed two distinct robotic platforms with specialized software stacks:

The UGV ("Bob")

  • Hardware: An omnidirectional base equipped with a UR10e robotic arm and a custom magnetic gripper. A standout feature is its actuated storage system, which allows the robot to carry up to 10 large bricks at once, drastically reducing travel time.
  • Perception: Uses a 3D LiDAR (Velodyne VLP-16) on the wrist. The pipeline uses Iterative Closest Point (ICP) for rough alignment of the brick pile, followed by a fine-grained optimization that considers the physical constraints of the wall structure.
  • Planning: Employs a backtracking algorithm to find the time-optimal build order, ensuring the robot places bricks in a sequence that minimizes base movement.

The UAV ("Lofty")

  • Hardware: A DJI Matrice 600 customized with a unique compliant magnetic gripper. The gripper uses telescopic carbon fiber rods that allow for horizontal and rotational "give" during the approach but become rigid once the brick is lifted.
  • Perception: Relies on three Intel RealSense D435 RGB-D cameras. It uses a color-based segmentation method (looking for white patches on colored bricks) and a Perspective-n-Point (PnP) solver for real-time 6D pose estimation.
  • Control: Uses a "cone of descent" strategy for grasping and placement, ensuring the drone stays stable even in windy conditions.

Datasets

The paper does not use standard public datasets. Instead, the systems were evaluated on:

  • MBZIRC 2020 Finals: Real-world performance in a 40x50m outdoor arena in Abu Dhabi.
  • Custom Lab Experiments: 1,000 randomly generated wall blueprints were used to test the build-order optimization, and controlled "pick-and-place" trials were used to measure precision.

Key Insights and Ideas

  • Mechanical Intelligence over Software Complexity: The UAV’s telescopic gripper is a masterclass in "passive compliance." By allowing the gripper to move freely when pushed against a brick, the researchers simplified the control problem—the drone doesn't need perfect flight precision to achieve a perfect grasp.
  • Storage is Efficiency: Most construction robots move one brick at a time. By adding an actuated "magazine" to the UGV, the team transformed the task from a series of long hauls into a high-speed assembly process.
  • Context-Aware Perception: The UGV’s perception system doesn't just look for "boxes"; it uses the known wall blueprint as a prior. It "knows" where bricks should be, which makes the ICP (Iterative Closest Point) algorithm much more robust against noise and occlusions.

Key Results

  • Competition Success: The team achieved 2nd place overall in Challenge 2 of MBZIRC 2020.
  • UGV Precision: In lab tests, the UGV achieved an end-to-end manipulation precision of approximately 1.3–1.9 cm in translation and under 1.3° in rotation, which is sufficient for stacking bricks without manual intervention.
  • Optimization Gains: The backtracking algorithm reduced the number of required robot base positions from 5.5 to 5.0 on average and saved 2.3 meters of driving per wall segment compared to a greedy approach.
  • UAV Reliability: During the competition, the UAV attempted 132 picks with a 16.7% success rate. While seemingly low, this was highly competitive given the extreme difficulty of aerial manipulation in wind.

Limitations and Future Work

The authors are candid about the "reality gap" encountered during the competition:

  • Hardware Fragility: The UGV suffered from gripper issues and difficulty driving on the arena's internal slopes, which weren't fully replicated in lab tests.
  • Testing Time: Many design decisions were made late in the process, leaving insufficient time for "full-stack" testing under competition constraints.
  • Future Directions: The team aims to improve UGV navigation on uneven terrain and further integrate UAV-UGV cooperation, such as having the UGV act as a mobile "refueling" or "reloading" station for the drones.

Summary updated

Abstract

Constructing large structures with robots is a challenging task with many potential applications that requires mobile manipulation capabilities. We present two systems for autonomous wall building that we developed for the Mohamed Bin Zayed International Robotics Challenge 2020. Both systems autonomously perceive their environment, find bricks, and build a predefined wall structure. While the UGV uses a 3D LiDAR-based perception system which measures brick poses with high precision, the UAV employs a real-time camera-based system for visual servoing. We report results and insights from our successful participation at the MBZIRC 2020 Finals, additional lab experiments, and discuss the lessons learned from the competition.
BibTeX
@inproceedings{lenz2020autonomous,
  author = {Christian Lenz and Max Schwarz and Andre Rochow and Jan Razlaw and Arul Selvam Periyasamy and Michael Schreiber and Sven Behnke},
  title = {{Autonomous Wall Building with a UGV- UAV Team at MBZIRC 2020}},
  booktitle = {SSRR},
  year = {2020},
  url = {https://openreview.net/forum?id=dSjgJ7elqo-x},
}

Loading related content...

Figure