Grounded Semantic Role Labelling from Synthetic Multimodal Data for Situated Robot Commands

Claudiu Daniel Hromei, Antonio Scaiella, Danilo Croce, Roberto BasiliEMNLP 2025

TL;DR

This paper introduces a novel framework for Grounded Semantic Role Labelling (G-SRL) that bridges natural language robot commands with visual perception. They achieve this by developing an automated pipeline to generate high-quality synthetic multimodal training data, which significantly improves the performance of vision-language models in understanding situated commands.

What Problem Does This Paper Address?

Robots operating in human-centric environments need to understand natural language commands that are grounded in their current perceptual context. Traditional symbolic parsers struggle with the dynamic and complex nature of real-world environments. While existing approaches have made strides in mapping language to actions, they often lack robust perceptual grounding or rely on costly, manually curated datasets. This paper addresses the challenge of creating a flexible and scalable system that can interpret robot commands by linking linguistic elements to concrete visual entities and spatial relationships, adapting to the robot's perception of the environment.

Methods

The core of the paper is a novel Grounded Semantic Role Labelling (G-SRL) framework. This framework combines frame semantics (predicate-argument structures) with explicit perceptual grounding. The key innovation lies in their automated pipeline for generating synthetic multimodal training data. This pipeline involves:

  1. Constraint Extraction: Deriving semantic constraints (entities, spatial relations, object states) from natural language commands and their logical forms.
  2. Prompt Generation: Using Large Language Models (LLMs) to create rich, semantically controlled textual prompts for image generation, incorporating the extracted constraints.
  3. Image Generation: Employing diffusion-based image synthesis models to create photorealistic scenes that visually express the semantic constraints.
  4. Constraint Validation: A crucial step that uses object detection (GroundingDINO) and visual question answering (MiniCPM-V) to ensure the generated images are semantically accurate and prevent hallucinations.

This pipeline generates diverse training data without manual scene design, mimicking real-world ambiguities. They also fine-tune Vision-Language Models (VLMs) like MiniCPM-V as perceptual semantic parsers.

Datasets

The primary dataset used is the HuRIC corpus, which contains frame-semantic annotations for robot commands. The authors augment this corpus by generating over 11,000 image-command-logical form triplets using their automated pipeline. This generated dataset is referred to as the Complete Dataset. A subset of this, manually validated for quality, is called the Validated Dataset.

Key Insights and Ideas

  • Synthetic Data for Robust Grounding: Generating diverse, contextually rich synthetic data, even with some noise, significantly improves a model's ability to ground semantic roles in visual perception.
  • Dynamic Frame Adaptation: The G-SRL framework allows for dynamic adaptation of frame structures based on visual context, mirroring how robots perceive and interpret commands in real-time.
  • Automated Data Generation is Scalable: The proposed pipeline offers a cost-effective and scalable solution for creating high-quality multimodal datasets, overcoming the limitations of manual annotation and traditional simulation environments.
  • Validation is Crucial: The rigorous validation process using object detection and VQA is essential for ensuring the semantic accuracy of synthetic data and preventing models from learning from misleading examples.

Key Results

  • Models fine-tuned on their automatically generated data perform comparably to those trained on manually curated instances, significantly reducing cost and effort.
  • The proposed validation strategy reliably filters noise, yielding high-quality, semantically consistent images.
  • Fine-tuning MiniCPM-V on their dataset led to substantial performance improvements, with frame prediction exceeding 97% F1 and grounding metrics also showing significant gains.
  • Increasing the diversity of training data (using Top-k ranked images) consistently improved grounding accuracy (IoU), reaching nearly 64.66% on Top-10.
  • Grounding errors, particularly misgenerated bounding boxes, remain the most frequent error type, but visual diversity helps reduce these.

Limitations and Future Work

  • Domain Specificity: The current dataset is restricted to domestic environments, limiting immediate applicability to other domains like industrial settings.
  • Synthetic Data: All images are synthetic. While human annotators ensured perceived realism, future work could explore topological consistency across multiple scenes.
  • 2D Representations: The current setup uses 2D renderings, lacking 3D geometry or depth information, which could be beneficial for more complex spatial reasoning.
  • Frame Coverage: The dataset's frame set is representative but not exhaustive; future work could extend coverage.
  • Language Specificity: The dataset is grounded in English and reflects a specific cultural setting; multilingual and cross-cultural extensions are desirable.
  • Future Work: Exploring autonomous inference of visual constraints from language, aligning with broader semantic formalisms like AMR, and training on 3D simulation environments are promising directions. The ultimate goal is to enable robots to interpret natural instructions by reasoning about both what they see and what they should expect to see.

Summary updated

BibTeX
@inproceedings{hromei2025grounded,
  author = {Claudiu Daniel Hromei and Antonio Scaiella and Danilo Croce and Roberto Basili},
  title = {{Grounded Semantic Role Labelling from Synthetic Multimodal Data for Situated Robot Commands}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure