RoboOmni: Proactive Robot Manipulation in Omni-modal Context

Siyin Wang, Jinlan Fu, Feihong Liu, Xinzhe He, Huangxuan Wu, Junhao Shi, Kexin Huang, Zhaoye Fei, Jingjing Gong, Zuxuan Wu, Yu-Gang Jiang, See-Kiong Ng, Tat-Seng Chua, Xipeng Qiu

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have driven rapid progress in Vision-Language-Action (VLA) models for robotic manipulation. Although effective in many scenarios, current approaches largely rely on explicit instructions, whereas in real-world interactions, humans rarely issue instructions directly. Effective collaboration requires robots to infer user intentions proactively. In this work, we introduce cross-modal contextual instructions, a new setting where intent is derived from spoken dialogue, environmental sounds, and visual cues rather than explicit commands. To address this new setting, we present RoboOmni, a Perceiver-Thinker-Talker-Executor framework based on end-to-end omni-modal LLMs that unifies intention recognition, interaction confirmation, and action execution. RoboOmni fuses auditory and visual signals spatiotemporally for robust intention recognition, while supporting direct speech interaction. To address the absence of training data for proactive intention recognition in robotic manipulation, we build OmniAction, comprising 140k episodes, 5k+ speakers, 2.4k event sounds, 640 backgrounds, and six contextual instruction types. Experiments in simulation and real-world settings show that RoboOmni surpasses text- and ASR-based baselines in success rate, inference speed, intention recognition, and proactive assistance.

Introduction

Current Vision-Language-Action (VLA) models typically operate under a "Simon Says" paradigm, requiring explicit, direct commands to perform tasks. However, in natural human environments, instructions are often implicit, embedded within multi-party dialogues, emotional tones, or environmental sounds. For instance, a robot overhearing a user say "I'm thirsty" while looking at a fridge should ideally infer the need for a drink without a direct command.

Existing systems often rely on cascaded Automatic Speech Recognition (ASR) pipelines, which transcribe speech to text. This approach suffers from significant information loss—discarding paralinguistic cues like tone or urgency—and high latency, which hinders real-time proactive assistance. To bridge this gap, we introduce RoboOmni, an end-to-end omni-modal framework designed for cross-modal contextual instructions.

RoboOmni shifts from direct commands to inferring intent from complex, multi-modal contexts including speech, sound events, and visual cues.

The RoboOmni Framework

RoboOmni utilizes a Perceiver-Thinker-Talker-Executor architecture built on an omni-modal Large Language Model (LLM) backbone (Qwen2.5-Omni).

  • Perceiver: This module fuses visual observations and raw audio signals into a unified embedding space. By processing audio directly, the model "hears" environmental sounds and vocal nuances that ASR would miss.
  • Thinker: The core reasoning engine that performs intent recognition and high-level planning based on multi-modal inputs.
  • Talker: A hierarchical speech generation module that enables the robot to engage in natural dialogue, such as asking for confirmation before acting on an inferred intent.
  • Executor: This module maps the Thinker's plans to 7-DoF motor actions using a specialized action tokenizer, ensuring smooth and continuous robot control.

The OmniAction Dataset

A major hurdle in training proactive robots is the lack of suitable data. To address this, we developed OmniAction, a large-scale dataset specifically designed for contextual instruction following.

The OmniAction pipeline generates diverse training data through dialogue synthesis, interaction extension, and auditory realization.

OmniAction consists of 140,000 episodes covering 112 skills and 748 objects. It features over 5,000 unique speaker timbres and 2,400 event sounds. The dataset categorizes instructions into six complex types: Sentiment Cues (emotion), Overlapping Voices (crosstalk), Non-Verbal Cues (environmental sounds), Identity Cues (speaker recognition), and Dyadic/Triadic Dialogue (multi-person context).

Experimental Evaluation

RoboOmni was evaluated on the OmniAction-LIBERO simulation benchmark and in real-world settings using a WidowX 250S robot arm.

  • Performance: RoboOmni achieved an 85.6% success rate, significantly outperforming the state-of-the-art NORA baseline (25.9%).
  • Efficiency: By bypassing the ASR bottleneck, RoboOmni reduced latency to 0.49x that of cascaded pipelines, effectively doubling inference speed.
  • Generalization: Pretraining on OmniAction proved crucial. A pretrained model reached 90% accuracy in new tasks within 2,000 steps, whereas a model trained from scratch failed to reach 30% even after 20,000 steps.

Conclusion

RoboOmni demonstrates that end-to-end omni-modal processing is essential for creating truly proactive robotic assistants. By integrating auditory and visual signals directly, robots can move beyond literal command-following to more nuanced, human-like collaboration. Future work will focus on incorporating more "in-the-wild" organic data and improving long-horizon reasoning for complex household tasks.

Summary updated

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have driven rapid progress in Vision-Language-Action (VLA) models for robotic manipulation. Although effective in many scenarios, current approaches largely rely on explicit instructions, whereas in real-world interactions, humans rarely issue instructions directly. Effective collaboration requires robots to infer user intentions proactively. In this work, we introduce cross-modal contextual instructions, a new setting where intent is derived from spoken dialogue, environmental sounds, and visual cues rather than explicit commands. To address this new setting, we present RoboOmni, a Perceiver-Thinker-Talker-Executor framework based on end-to-end omni-modal LLMs that unifies intention recognition, interaction confirmation, and action execution. RoboOmni fuses auditory and visual signals spatiotemporally for robust intention recognition, while supporting direct speech interaction. To address the absence of training data for proactive intention recognition in robotic manipulation, we build OmniAction, comprising 140k episodes, 5k+ speakers, 2.4k event sounds, 640 backgrounds, and six contextual instruction types. Experiments in simulation and real-world settings show that RoboOmni surpasses text- and ASR-based baselines in success rate, inference speed, intention recognition, and proactive assistance.
BibTeX
@inproceedings{wang2026roboomni,
  author = {Siyin Wang and Jinlan Fu and Feihong Liu and Xinzhe He and Huangxuan Wu and Junhao Shi and Kexin Huang and Zhaoye Fei and Jingjing Gong and Zuxuan Wu and Yu-Gang Jiang and See-Kiong Ng and Tat-Seng Chua and Xipeng Qiu},
  title = {{RoboOmni: Proactive Robot Manipulation in Omni-modal Context}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=OJh7oBCYhL},
}
Figure