Foundation Models in Robotics: A Comprehensive Review of Methods, Models, Datasets, Challenges and Future Research Directions

Aggelos Psiris, Vasileios Argyriou, Evangelos K. Markakis, Panagiotis Sarigiannidis, Efstratios Gavves, Kostas Bekris, Arash Ajoudani, Georgios Th. PapadopoulosarXiv 2026 · arXiv

The field of robotics is currently experiencing its "GPT moment." For decades, robots were specialists—programmed or trained to do one specific thing in a controlled environment. If you wanted a robot to move from a factory floor to a kitchen, you essentially had to start from scratch.

A new comprehensive review paper, Foundation Models in Robotics, explores how the same "Foundation Model" (FM) revolution that gave us ChatGPT is now creating generalist robotic agents capable of reasoning, planning, and acting across diverse, unpredictable environments.

TL;DR

This paper provides a holistic roadmap of how Large Language Models (LLMs) and Vision Models are being transformed into "Vision-Language-Action" (VLA) models for robotics. It moves beyond simple task-specific AI to explore how robots can now use internet-scale knowledge to perform complex, long-horizon tasks across different physical bodies (embodiments).

What Problem Does This Paper Address?

Traditional robot learning is "brittle." If a robot is trained to pick up a red ball in a lab, it might fail if the ball is blue or the lighting changes. This is because robots historically lacked common-sense reasoning and generalization.

The authors tackle the challenge of how to integrate massive, "internet-scale" datasets (text and images) with the messy, high-frequency, physical reality of robot hardware. They categorize the massive explosion of recent research to help engineers understand which architectures (like Transformers vs. State-Space Models) and which learning paradigms (like Imitation Learning vs. Reinforcement Learning) actually work for real-world deployment.

Methods: The Five Phases of Evolution

The authors delineate the progress of the field through five distinct research phases, showing a clear trajectory from modular "brains" to integrated "bodies."

Figure 1: Main phases in robotic FM research and key/milestone works.
  1. Phase 1 (2018–2021): Using "frozen" NLP and CV models just for perception (e.g., "find the cup").
  2. Phase 2 (2021–2022): Grounded planning, where models link visual inputs to language commands to create step-by-step instructions.
  3. Phase 3 (2022–2023): The birth of VLA (Vision-Language-Action). Models like RT-1 and PaLM-E began outputting direct robot actions (tokens) instead of just text.
  4. Phase 4 (2023–2024): Focus on memory and "Web-to-robot" transfer, allowing robots to learn from human videos on YouTube.
  5. Phase 5 (2024–Present): The current frontier—multi-sensory generalization (touch, force, sound) and real-world deployment on humanoid platforms.

Datasets

Data is the biggest bottleneck in robotics. Unlike text, robot data is expensive and "physically grounded." The paper highlights several key dataset families:

  • Cross-Embodiment Corpora: The landmark Open X-Embodiment dataset, which aggregates over 1 million trajectories from 22 different robot platforms.
  • Egocentric Human Video: Datasets like Ego4D, which provide first-person video of humans performing daily tasks, helping robots learn "human priors."
  • Simulation Suites: Tools like AI2-THOR and Isaac Gym that allow for safe, high-speed training before moving to real hardware.

Key Insights and Ideas

  • The Semantic-Physical Gap: An LLM might know that "to make coffee, you need a mug," but it doesn't know how much force is required to grip that specific mug without breaking it. Bridging this "grounding" gap is the core challenge of modern robotics.
  • From Transformers to SSMs: While Transformers dominate reasoning, the paper notes a shift toward State-Space Models (SSMs) like Mamba. Why? Because Transformers have "quadratic complexity"—they get exponentially slower as the robot's "memory" grows. SSMs offer linear scaling, which is vital for real-time, 50Hz control loops.
  • The "Final Frontier" of Haptics: Most current models are "vision-heavy." The authors argue that true physical intelligence requires a "Foundation Model for Touch," integrating tactile and force feedback to achieve human-level dexterity.

Key Results

The paper is a systematic review rather than an empirical study, but it identifies critical trends:

  • Generalization is real: Models trained on diverse data (Phase 3 and 4) show a massive jump in "zero-shot" performance—the ability to handle objects they have never seen before.
  • Latency is the enemy: Large models (like PaLM-E with 562B parameters) provide incredible reasoning but suffer from high inference latency, making them difficult to use for fast, reactive motions.
  • Human-Robot Interaction (HRI): FMs have fundamentally changed HRI, moving from rigid "if-then" commands to natural language dialogue where robots can ask for clarification (e.g., "Do you mean the red mug or the blue one?").

Limitations and Future Work

Despite the hype, several "open problems" remain:

  1. Data Scarcity for Failures: We have lots of data on robots succeeding, but very little on robots failing. Models need to see "long-tail" failure scenarios to learn how to recover safely.
  2. Heterogeneous Action Spaces: We still don't have a "universal controller." A command for a drone is mathematically different from a command for a robotic arm. Developing "embodiment-agnostic" action spaces is a major future goal.
  3. Safety and Verification: FMs can "hallucinate" actions that are physically dangerous. The authors suggest that future work must integrate formal mathematical safety guardrails directly into the neural network's reasoning loop.

For researchers looking to enter the field, this paper serves as a vital "state of the union," mapping out exactly where the "brain" of AI meets the "body" of robotics.

Summary updated

Abstract

Over the recent years, the field of robotics has been undergoing a transformative paradigm shift from fixed, single-task, domain-specific solutions towards adaptive, multi-function, generalpurpose agents, capable of operating in complex, open-world, and dynamic environments. This tremendous advancement is primarily driven by the emergence of Foundation Models (FMs), i.e., large-scale neural-network architectures trained on massive, heterogeneous datasets that provide unprecedented capabilities in multi-modal understanding and reasoning, long-horizon planning, and cross-embodiment generalization. In this context, the current study provides a holistic, systematic, and in-depth review of the research landscape of FMs in robotics. In particular, the evolution of the field is initially delineated through five distinct research phases, spanning from the early incorporation of Natural Language Processing (NLP) and Computer Vision (CV) models to the current frontier of multi-sensory generalization and real-world deployment. Subsequently, a highly-granular taxonomic investigation of the literature is performed, examining the following key aspects: a) the employed FM types, including LLMs, VFMs, VLMs, and VLAs, b) the underlying neural-network architectures, c) the adopted learning paradigms, d) the different learning stages of knowledge incorporation, e) the major robotic tasks, and f) the main real-world application domains. For each aspect, comparative analysis and critical insights are provided. Moreover, a report on the publicly available datasets used for model training and evaluation across the considered robotic tasks is included. Furthermore, a hierarchical discussion on the current open challenges and promising future research directions in the field is incorporated.
BibTeX
@misc{psiris2026foundation,
  author = {Aggelos Psiris and Vasileios Argyriou and Evangelos K. Markakis and Panagiotis Sarigiannidis and Efstratios Gavves and Kostas Bekris and Arash Ajoudani and Georgios Th. Papadopoulos},
  title = {{Foundation Models in Robotics: A Comprehensive Review of Methods, Models, Datasets, Challenges and Future Research Directions}},
  year = {2026},
  eprint = {2604.15395},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.15395},
}

Loading related content...

Figure