TL;DR
FlipAnything introduces an agentic framework where an LLM-based "coding agent" autonomously develops and refines robotic manipulation strategies through iterative self-learning in simulation. By evolving a structured, graph-based policy rather than training a black-box neural network, the system improved its success rate on a diverse 24-object flipping benchmark from 24.7% to 95.0%, demonstrating zero-shot transfer to real-world hardware.
What Problem Does This Paper Address?
Flipping an object 180°—like turning a coin from heads to tails—is a deceptive task. While it sounds simple, it is a fundamental manipulation skill that varies wildly depending on the object's geometry. A hammer might require a bimanual handover in mid-air, while a flat multimeter might need to be pushed to the edge of a table and toppled.
Traditional approaches struggle here: manual coding for every possible object is unscalable, and end-to-end reinforcement learning (RL) often lacks the structured reasoning required to switch between distinct strategies (e.g., pushing vs. grasping). The authors tackle the challenge of creating a single, robust policy capable of autonomously selecting and executing the right flipping strategy for diverse, unseen objects.
Methods
The core of the paper is Agentic Self-Learning powered by the Graph-as-Policy (GaP) harness. Instead of training weights in a neural network, the system uses an LLM (Claude Fable 5.1) to write and refine executable code.
The process follows a four-stage self-learning loop:
- Generate: The agent writes an initial graph-based policy (starting with simple bimanual handovers).
- Evaluate: The policy is tested in RoboSimStudio (a MuJoCo-based simulator) across 24 objects with randomized starting positions.
- Analyze: The agent receives feedback in the form of success rates, execution traces, and failure diagnostics (e.g., "lost contact," "infeasible grasp").
- Revise: The agent modifies the code to add new strategies (like "stand-and-topple"), refine perception, or add recovery logic.
The resulting policy is a structured graph that routes observations through plan selection, strategy execution (with reactive "reflexes"), and a final judgment phase.
Datasets
The authors introduced the FlipAnything benchmark, consisting of 24 diverse CAD models of everyday objects, including tools (hammers, pliers), electronics (multimeters, breadboards), and household items (tape, safety glasses).
Evaluation involved 360 episodes per iteration (15 per object). For real-world validation, they tested the simulation-learned code zero-shot on three objects not seen during the learning phase: a clamp, blue tape, and a foldable helix screwdriver.
Key Insights and Ideas
- Code as a Generalizable Policy: Unlike black-box models, structured code allows the agent to explicitly reason about geometry. It can decide that a "thin" object needs a different approach than a "wide" one, making the policy highly interpretable and adaptable.
- Strategy Diversity is Key: The agent didn't just get better at one motion; it discovered new modes of manipulation. It moved from simple handovers to complex behaviors like "edge sliding" (pushing an object to the table edge to get a better grip) and "pinch-standing."
- Reflexes and Recovery: A major performance jump came from the agent adding "reflexes"—closed-loop behaviors like in-hand pose correction and re-grasping when a slip is detected.
- Simulation-to-Reality (Sim2Real): Because the policy is based on high-level geometric primitives and spatial reasoning rather than raw pixel-to-torque mapping, it transferred surprisingly well to physical robots without any real-world fine-tuning.
Key Results
- Performance Leap: The success rate started at a meager 24.7% (handover only) and climbed to 80.6% after the agent added toppling and pushing strategies. By the final iteration (v36), which included robust sensing and recovery, the success rate reached 95.0%.
- Object-Specific Success: The system achieved perfect 15/15 scores on complex items like multimeters and lock nuts. Even challenging "thin" objects like surgical scissors improved from 0/15 to 13/15.
- Real-World Transfer: In physical trials, the agent succeeded in 2/3 trials for the clamp and 3/3 for the tape. The failure on the complex helix screwdriver (1/3) highlighted the remaining gap between simulation and reality.
Limitations and Future Work
While impressive, the authors note that the system still relies on having CAD models for the simulation phase. The real-world performance, while promising, is not yet at the 95% level seen in simulation, particularly for objects with complex mechanical linkages (like the foldable screwdriver).
Future work will focus on improving generalization to entirely unseen geometries and further closing the Sim2Real gap by incorporating more diverse physical failure modes into the training loop.
Summary updated