CLIPUNetr: Assisting Human-robot Interface for Uncalibrated Visual Servoing Control with CLIP-driven Referring Expression Segmentation

Chen Jiang, Yuchen Yang, Martin Jägersand

Abstract

The classical human-robot interface in uncalibrated image-based visual servoing (UIBVS) relies on either human annotations or semantic segmentation with categorical labels. Both methods fail to match natural human communication and convey rich semantics in manipulation tasks as effectively as natural language expressions. In this paper, we tackle this problem by using referring expression segmentation, which is a prompt-based approach, to provide more in-depth information for robot perception. To generate high-quality segmentation predictions from referring expressions, we propose CLIPUNetr - a new CLIP-driven referring expression segmentation network. CLIPUNetr leverages CLIP's strong vision-language representations to segment regions from referring expressions, while utilizing its “U-shaped” encoder-decoder architecture to generate predictions with sharper boundaries and finer structures. Furthermore, we propose a new pipeline to integrate CLIPUNetr into UIBVS and apply it to control robots in real-world environments. In experiments, our method improves boundary and structure measurements by an average of 120% and can successfully assist real-world UIBVS control in an unstructured manipulation environment.

Introduction

In robotics, Uncalibrated Image-Based Visual Servoing (UIBVS) allows for robot control based on visual feedback without a precise mathematical model of the environment. However, traditional UIBVS interfaces are limited by manual target selection or rigid, pre-defined semantic categories. These methods struggle with the nuance of natural human instructions, such as "grab the handle of the cup" or "find something to contain liquid."

This research introduces CLIPUNetr to bridge this gap. By utilizing Referring Expression Segmentation (RES), the system enables robots to interpret arbitrary natural language prompts, allowing for more flexible and intuitive human-robot interaction.

Methodology: The CLIPUNetr Architecture

CLIPUNetr combines the semantic power of large-scale vision-language models with the structural precision required for physical manipulation.

Model Design

The architecture utilizes a pre-trained CLIP (Contrastive Language-Image Pre-training) Vision Transformer (ViT) as its backbone. While standard CLIP-based models often produce low-resolution, blurry masks, CLIPUNetr incorporates a U-shaped encoder-decoder structure. This design extracts features at multiple scales and employs skip connections to preserve fine-grained structural details, which are essential for precise robotic grasping.

The CLIPUNetr architecture integrates a CLIP visual encoder with a multi-scale U-shaped decoder and FiLM-based text conditioning.

Language Integration and Control

Text prompts are processed by CLIP’s text encoder and injected into the visual features using Feature-wise Linear Modulation (FiLM). This mechanism directs the visual decoder to prioritize regions relevant to the user's instruction. The resulting high-fidelity segmentation mask is then processed via Principal Component Analysis (PCA) to derive geometric constraints (points and lines), which are fed into a Broyden-updated Jacobian controller to execute robot movement.

Experiments and Results

The model was evaluated on the PhraseCut dataset and a custom-annotated version of the UMD affordance dataset (UMD+GT), featuring 39 diverse prompt templates ranging from simple object names to complex functional descriptions.

Performance Metrics

CLIPUNetr demonstrated a significant leap in performance over existing baselines like CLIPSeg. Specifically, the model achieved a 120% average improvement in boundary and structure measurements ($S_m$). Unlike previous transformer-based models that suffer from "checkerboard artifacts," CLIPUNetr produces smooth, continuous probability maps with sharp edges.

Qualitative comparison showing CLIPUNetr's ability to generate sharper, more accurate masks compared to CLIPSeg across various prompts.

Real-World Validation

In online trials using a Kinova Gen3 robot, the CLIPUNetr-driven interface achieved a 100% success rate in grasping tasks. This outperformed classical manual-tracking interfaces, which showed a 20% failure rate when tasked with manipulating complex items like food in unstructured environments.

Conclusion

CLIPUNetr represents a significant advancement in human-robot interfaces, moving beyond rigid labels toward functional, language-driven perception. By prioritizing structural accuracy and boundary precision, the system enables reliable robotic manipulation in real-world settings. Future work will focus on optimizing inference speed for real-time applications and extending the system's capabilities to more cluttered, complex environments.

Summary updated

Abstract

The classical human-robot interface in uncalibrated image-based visual servoing (UIBVS) relies on either human annotations or semantic segmentation with categorical labels. Both methods fail to match natural human communication and convey rich semantics in manipulation tasks as effectively as natural language expressions. In this paper, we tackle this problem by using referring expression segmentation, which is a prompt-based approach, to provide more in-depth information for robot perception. To generate high-quality segmentation predictions from referring expressions, we propose CLIPUNetr - a new CLIP-driven referring expression segmentation network. CLIPUNetr leverages CLIP's strong vision-language representations to segment regions from referring expressions, while utilizing its ``U-shaped'' encoder-decoder architecture to generate predictions with sharper boundaries and finer structures. Furthermore, we propose a new pipeline to integrate CLIPUNetr into UIBVS and apply it to control robots in real-world environments. In experiments, our method improves boundary and structure measurements by an average of 120% and can successfully assist real-world UIBVS control in an unstructured manipulation environment.
BibTeX
@misc{jiang2023clipunetr,
  author = {Chen Jiang and Yuchen Yang and Martin Jägersand},
  title = {{CLIPUNetr: Assisting Human-robot Interface for Uncalibrated Visual Servoing Control with CLIP-driven Referring Expression Segmentation}},
  year = {2023},
  eprint = {2309.09183},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.09183},
}
Figure