Eye-In-Finger: Smart Fingers for Delicate Assembly and Disassembly of LEGO

Zhenran Tang, Ruixuan Liu, Changliu Liu

TL;DR

This paper introduces "Eye-in-Finger" (EiF), a novel end-effector design that embeds a camera directly into the tool tip for precise, occlusion-free robotic manipulation. EiF significantly improves accuracy and robustness in delicate tasks like LEGO assembly by enabling real-time, fine-grained visual feedback, overcoming limitations of traditional vision systems.

What Problem Does This Paper Address?

Robotic assembly of small, tightly toleranced objects, such as LEGO bricks, presents significant challenges. Traditional vision-based approaches often struggle with occlusions caused by the robot's own end-effector or the evolving assembly. Furthermore, achieving the sub-millimeter accuracy required for these tasks is difficult with external cameras, which are prone to calibration errors and lack the necessary local perception. While tactile sensing offers high resolution, it can be costly, fragile, and unsuitable for tasks requiring rigid contact. This paper addresses the need for a low-cost, high-precision, and occlusion-free sensing solution integrated directly at the point of manipulation.

Methods

The core innovation is the "Eye-in-Finger" (EiF) hardware-software co-design.

  • Hardware: EiF integrates a low-cost endoscope camera with built-in LEDs directly into the tool tip, behind a hollow section that securely snaps onto LEGO brick knobs. This design ensures the camera has a clear view of the target brick as long as it's within reach, minimizing occlusion and maintaining a compact form factor.
  • Pose Estimation: For partially occluded LEGO bricks, a novel pose estimation method is employed. This involves:
  • Segmenting the brick using a YOLOv8-seg model.
  • Identifying bounding lines for the closest pair of knobs.
  • Optimizing a circle fit to the knob's mask, constrained by these bounding lines and a cost function that balances mask coverage and estimated circle area. This yields precise knob centers.
  • Tilt Estimation: To address structural weaknesses from slight tilts, a reflection-based method is used. The camera analyzes the reflection of its own LED ring off the smooth LEGO brick surface. By tracking the shift of the reflection, pitch and roll angles are accurately estimated, even for small tilts.
  • Manipulation Process: The system orchestrates pick-and-place operations using a visual servoing pipeline. It iteratively refines the robot's position based on the estimated pose and tilt offsets until a predefined accuracy threshold is met, enabling robust error correction during dynamic assembly.

Datasets

Not explicitly discussed in the paper. The authors mention fine-tuning YOLOv11 on collected images with reflections for tilt detection and using a YOLOv8-seg model for segmentation. The primary evaluation is performed on LEGO assembly and disassembly tasks using a physical robot setup.

Key Insights and Ideas

  • Perception at the Point of Contact is Crucial: Embedding vision directly into the end-effector fundamentally changes the perception paradigm, moving from global to local, high-resolution sensing where it matters most.
  • LEGO as a Challenging Benchmark: The tight tolerances and interlocking nature of LEGO bricks make it an excellent, yet demanding, testbed for evaluating fine-grained robotic manipulation and sensing capabilities.
  • Reflection for Tilt Sensing: Leveraging the smooth surfaces of LEGO bricks and the integrated LED ring to detect subtle tilts is an elegant and low-cost solution for a critical aspect of structural integrity.
  • Robustness to Calibration Errors: The real-time feedback loop provided by EiF significantly reduces reliance on precise initial calibration, allowing for successful manipulation even with substantial errors.

Key Results

  • Measurement Accuracy: EiF achieved significantly higher measurement accuracy compared to a wrist-mounted Intel RealSense D405 camera:
  • 8.8-fold improvement in X/Y position accuracy.
  • 1.9-fold improvement in Yaw.
  • 13.6-fold improvement in Pitch.
  • 26.1-fold improvement in Roll.
  • Calibration Error Tolerance: The system demonstrated a remarkable increase in calibration error tolerance. While an open-loop system failed beyond 0.8 mm calibration error, EiF maintained 100% success rates for errors up to 2.0 mm, with success dropping only when occlusion became severe.
  • Teleoperation Efficiency: In teleoperation tasks, using the EiF view resulted in a 17% faster task completion time and a 19% reduction in positioning error compared to a traditional third-person view.

Limitations and Future Work

  • Limited Field of View: The camera's small field of view within the end-effector requires an estimated initial position of the target object. For broader applications, an additional global camera would be needed for high-level perception and guidance.
  • Future Integration: The authors plan to integrate EiF with a global camera to overcome the FOV limitation and enable more comprehensive perception. They also aim to incorporate learning-based manipulation policies for enhanced dexterity and broader task applicability.

Summary updated

BibTeX
@inproceedings{tang2025eye,
  author = {Zhenran Tang and Ruixuan Liu and Changliu Liu},
  title = {{Eye-In-Finger: Smart Fingers for Delicate Assembly and Disassembly of LEGO}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  doi = {10.1109/iros60139.2025.11246710},
}

Loading related content...

Figure