EC2: Emergent Communication for Embodied Control

Yao Mu, Shunyu Yao, Mingyu Ding, Ping Luo, Chuang Gan

Bridging the Modality Gap in Embodied AI

In the field of embodied AI, the goal is to develop robotic agents capable of learning new tasks rapidly from minimal input—a paradigm known as few-shot imitation learning. Typically, these agents are guided by either a video demonstration or a high-level natural language instruction. However, a significant "modality gap" persists between these two forms of guidance. Video demonstrations are rich in low-level perceptual details but lack abstract structure, while natural language provides symbolic generalization but misses the fine-grained motor control specifics.

Existing methods often attempt to bridge this gap using contrastive learning to force video and language representations into a shared embedding space. The authors of EC² (Emergent Communication for Embodied Control) argue that this alignment is overly restrictive, as it discards the unique, complementary information inherent to each modality. Instead, they propose using an unsupervised "emergent language" as a bridge.

The EC2 Framework

The EC² framework operates in three distinct phases designed to synthesize the strengths of both symbolic language and perceptual video data.

The EC2 framework: (a) Emergent Communication generation, (b) Language Model pre-training, and (c-d) downstream instruction following.
  1. Emergent Communication (EC) Generation: The process begins with a "referential game" between a Speaker and a Listener network. The Speaker translates a video trajectory into a sequence of discrete tokens (emergent language). The Listener must then identify the original video from a set of distractors using only these tokens. This setup forces the emergent language to capture the fine-grained, task-relevant details of the video.
  2. Pre-training the Language Model: A GPT-like architecture is trained on a trajectory completion task. The model learns to predict masked segments of a latent video trajectory, prompted by either emergent language tokens or natural language captions. This dual-prompting strategy teaches the model to map both types of "language" to the same underlying robotic motions.
  3. Few-shot Policy Learning: For downstream tasks, the pre-trained language model is frozen. A lightweight policy network is then trained using a small number of expert demonstrations (5–25). The model uses the provided instruction (video or text) as a prompt to generate features for action output.

Experimental Evaluation

The model was pre-trained on the LOReL dataset, featuring 3,000 episodes of a Franka Emika Panda robot. Evaluation was conducted across two challenging benchmarks: MetaWorld, which involves diverse manipulation tasks like hammering and pushing, and Franka Kitchen, a complex multi-object environment. To ensure robustness, the authors tested the system across three different camera viewpoints.

Performance and Insights

EC² demonstrates superior efficiency and success rates compared to established baselines such as R3M and BC-Z.

Overall success rate comparison across tasks, showing the proposed method (UniLearn/EC2) outperforming R3M and BC-Z baselines.

In MetaWorld, using only 10 demonstrations, EC² achieved a 76.0% success rate with video instructions, surpassing R3M's 71.8%. Similar gains were observed in the Franka Kitchen environment, where EC² showed a 5.2% improvement over R3M.

A key insight from the research is that emergent language is often more descriptive than standard captions. Qualitative analysis revealed that EC tokens accurately correspond to specific objects and actions (e.g., "drawer," "reach") more reliably than generic pre-trained models. Furthermore, the researchers found that the "fine-grainedness" of the emergent language—its ability to carry more information than natural language—is a primary driver of the system's success in downstream tasks.

Conclusion

EC² represents a shift away from simple modality alignment toward a more nuanced "bridge" approach. By leveraging emergent communication, the framework preserves the detail of video while gaining the symbolic structure of language. This results in robotic agents that are not only more successful at following instructions but also more efficient at learning from limited data. Future work will look toward scaling these methods to massive, real-world datasets and further decoding the "black box" of emergent language to improve human-robot collaboration.

Summary updated

Abstract

Embodied control requires agents to leverage multi-modal pre-training to quickly learn how to act in new environments, where video demonstrations contain visual and motion details needed for low-level perception and control, and language instructions support generalization with abstract, symbolic structures. While recent approaches apply contrastive learning to force alignment between the two modalities, we hypothesize better modeling their complementary differences can lead to more holistic representations for downstream adaption. To this end, we propose Emergent Communication for Embodied Control (EC^2), a novel scheme to pre-train video-language representations for few-shot embodied control. The key idea is to learn an unsupervised "language" of videos via emergent communication, which bridges the semantics of video details and structures of natural language. We learn embodied representations of video trajectories, emergent language, and natural language using a language model, which is then used to finetune a lightweight policy network for downstream control. Through extensive experiments in Metaworld and Franka Kitchen embodied benchmarks, EC^2 is shown to consistently outperform previous contrastive learning methods for both videos and texts as task inputs. Further ablations confirm the importance of the emergent language, which is beneficial for both video and language learning, and significantly superior to using pre-trained video captions. We also present a quantitative and qualitative analysis of the emergent language and discuss future directions toward better understanding and leveraging emergent communication in embodied tasks.
BibTeX
@inproceedings{mu2023ec,
  author = {Yao Mu and Shunyu Yao and Mingyu Ding and Ping Luo and Chuang Gan},
  title = {{EC2: Emergent Communication for Embodied Control}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00648},
}
Figure