Robotic manipulation has undergone a dramatic shift, moving from hand-engineered control loops to generalist policies trained via reinforcement learning, imitation learning, and large-scale sensorimotor pre-training. Yet, beneath these algorithmic triumphs lies a persistent structural bottleneck: the overwhelming majority of state-of-the-art models are evaluated on, and implicitly designed for, simple pick-and-place tasks.
When a robot picks up an object and drops it somewhere else, it treats the world as a sequence of quasi-static geometric states. The robot needs to know where the object is, how to shape its fingers to match the object's geometry, and where to move its arm. Once the grasp is secured, the physics of the task are largely frozen.
True general-purpose robots, however, must operate in a world of continuous, dynamic, and multi-stage interactions—such as using tools, manipulating cloth, pouring liquids, or assembling intricate machinery. Overcoming this task complexity limitation is the central frontier of modern robotics research.
The Pick and Place Plateau
The reliance on simplified manipulation tasks creates a false sense of security in robot learning. For example, self-supervised sequence prediction models have shown immense promise in learning physical world models. The Robot Pre-trained Transformer (RPT) developed by Ilija Radosavovic et al. in Robot Learning with Sensorimotor Pre-training demonstrates that masking and reconstructing multimodal "sensorimotor tokens" can yield policies that transfer seamlessly across hardware. However, the authors note that their evaluations remain confined to pick-and-place manipulations, leaving contact-rich interactions like tool use or cloth manipulation completely unexplored.
This plateau is deeply embedded in how the field benchmarks data efficiency and open-vocabulary generalization. Consider the following patterns across recent evaluation frameworks:
- Long-Horizon Benchmarks: The LAMBDA benchmark introduced by Ahmed Jaafar et al. in λ: A Benchmark for Data-Efficiency in Long-Horizon Indoor Mobile Manipulation Robotics pushes the boundaries of data efficiency in mobile robotics, yet its long-horizon tasks are still fundamentally composed of atomic pick-and-place sequences.
- Whole-Body Motion: The dataset and generation tools presented by Sixu Yan et al. in M${}^{3}$Bench: Benchmarking Whole-Body Motion Generation for Mobile Manipulation in 3D Scenes expand the spatial scale of robot simulation, but the underlying interactions do not yet venture past basic object retrieval and placement.
- Semantic Success Prediction: Even when models successfully align vision and natural language to predict whether a task succeeded—as shown by Miyu Goko et al. in Task Success Prediction for Open-Vocabulary Manipulation Based on Multi-Level Aligned Representations—the validation tasks are restricted to pick-and-place, meaning the models rarely have to reason about complex temporal sequences or multi-view physics.
- Data Retrieval: When borrowing data from large source datasets to enable few-shot learning, Amber Xie demonstrates in Data Retrieval with Importance Weights for Few-Shot Imitation Learning that importance-weighted retrieval significantly stabilizes policies. However, the author explicitly highlights that expanding these evaluations to highly dexterous tasks remains a vital next step to prove the generalizability of retrieval-based imitation.
The core issue is that pick-and-place acts as a geometric abstraction that shields the policy from the messy reality of physical forces.
The Physics of Continuous Contact and Dexterity
When a task transitions from simple grasping to dexterous or non-prehensile manipulation, the geometric assumptions break down. In-hand manipulation requires managing continuous slip, rolling contacts, and intricate force distributions across multiple fingers.
Tristan Luca Saidi et al. address this bottleneck in Sampling-based Exploration for Reinforcement Learning of Dexterous Manipulation, observing that existing manipulation frameworks frequently rely on simplified scenarios—such as using easy-to-handle objects or requiring external support surfaces—which severely restricts their real-world utility. Similarly, while Arthur Allshire et al. successfully train a policy to perform precise 6-DoF object reorientation in Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFinger, they emphasize that moving from spatial reposing to functional tool use remains an open research frontier.
[Pick-and-Place] -> Discrete Geometry (Grasp -> Move -> Release)
[Tool Use / Dexterity] -> Continuous Dynamics (Force Propagation -> Slip Control -> Multi-point Contact)
Non-prehensile manipulation—where objects are pushed, slid, or balanced rather than firmly grasped—introduces even higher dynamic uncertainty. For instance, Yuemin Mao et al. utilize acoustic sensing to predict and compensate for object sliding on a tray in Hearing the Slide: Acoustic-Guided Constraint Learning for Fast Non-Prehensile Transport, but extending this acoustic-guided learning to high-degree-of-freedom end effectors remains an open challenge.
When exploring unknown environments, Yitian Shi et al. leverage Bayesian uncertainty for online grasp learning in Uncertainty-driven Exploration Strategies for Online Grasp Learning, yet their framework is still bounded by prehensile mechanics. True complexity requires moving beyond collision avoidance entirely; Xinghao Zhu et al. argue in Allowing Safe Contact in Robotic Goal-Reaching: Planning and Tracking in Operational and Null Spaces that robots must safely exploit environmental contact to operate in cluttered spaces, though their current validation is limited to simple goal-reaching rather than multi-stage assembly. This limitation is mirrored in mobile manipulation; Kaiwen Jiang et al. achieve state-of-the-art 6D end-effector tracking in Learning Whole-Body Loco-Manipulation for Omni-Directional Task Space Pose Tracking With a Wheeled-Quadrupedal-Manipulator, but acknowledge that tracking a spatial trajectory does not inherently generalize to the intricate force objectives of complex manipulation.
Cognitive and Temporal Complexity
Beyond the immediate physics of contact, complex tasks introduce severe cognitive and perceptual challenges. A robot cannot simply react to what it currently sees; it must maintain a persistent mental model of its environment over time, especially when objects become occluded or change state out of view.
To address this, Yixuan Huang et al. introduce object-centric memory models (DOOM and LOOM) in Out of Sight, Still in Mind: Reasoning and Planning about Unobserved Objects with Video Tracking Enabled Memory Models to reason about hidden objects. Yet, even with advanced memory, the demonstrated tasks exclude complex articulated interactions like opening drawers.
When robots attempt to autonomously improve their skills, task complexity acts as the primary bottleneck. The SOAR framework by Zhiyuan Zhou et al. (Autonomous Improvement of Instruction Following Skills via Foundation Models) leverages foundation models for self-supervised data collection, but scaling this to dynamic tasks like pouring liquids or in-hand manipulation remains unsolved due to the fine motor control and deep physical understanding required.
This gap between low-level motor coordination and high-level cognitive goals is a long-standing challenge. As Georg Martius et al. demonstrate in Development of goal-oriented behavior in self-learning robots, guided self-organization can effectively produce basic locomotion and coordinated movements, but its applicability to intricate object manipulation or multi-step goal achievement remains unproven.
Constraints in Safety, Teleoperation, and Collaboration
As tasks scale in complexity, ensuring safety and seamless human-robot interaction becomes exponentially harder. In simple domains, safety constraints can be easily modeled or hardcoded. For instance, Parv kapoor et al. develop SafeDec in SafeDec: Constrained Decoding for Safe Autoregressive Generalist Robot Policies to enforce safety specifications at inference time without retraining, but the framework has primarily been evaluated on navigation rather than fine-grained manipulation.
In teleoperation and shared control, tasks are often heavily constrained to maintain stability. Early work by Weston B. Griffin et al. (Feedback Strategies for Telemanipulation with Shared Control of Object Handling Forces) demonstrated that shared force control improves delicate object handling, but the findings were restricted to a simple pick-carry-and-place task with uniform geometry. When modeling operator intent, Kevin Huang 0001 et al. utilize Hidden Markov Models for continuous authentication in Continuous Operator Authentication for Teleoperated Systems Using Hidden Markov Models, but note that their gesture segmentation is tailored to well-defined, simple tasks and struggles with unstructured real-world scenarios.
Even in specialized domains like surgical robotics, Young Gyun Kim et al. observe in A multimodal virtual vision platform as a next-generation vision system for a surgical robot that standard evaluation tasks (like ring transfer) fail to capture the full spectrum of complexities encountered in actual procedures.
Finally, human-robot collaboration faces a similar scalability wall. The Diffusion Co-policy framework introduced by Eley Ng et al. in Diffusion Co-Policy for Synergistic Human-Robot Collaborative Tasks successfully coordinates a shared table-carrying task by predicting human intent. However, the authors acknowledge that generalizing to complex co-manipulation requires integrating finer motor control and tactile feedback loops.
Moving Forward
The consensus across these diverse perspectives is clear: the algorithms we are building—whether they are autoregressive transformers, diffusion co-policies, or uncertainty-driven reinforcement learners—are outgrowing the tasks we use to test them. To break past the pick-and-place plateau, future research must design controllers, memory models, and benchmarks that treat continuous contact, tool use, and dynamic physics not as edge cases, but as the core definition of robotic manipulation.
Key Foundational Work
Feedback Strategies for Telemanipulation with Shared Control of Object Handling Forces (2005) — contributed 1 concept elements to this direction.
Vision-Based Guidance and Control of Robots in Projective Space (2000) — contributed 1 concept elements to this direction.