TL;DR
BURGER is a new benchmark that uses robotic drawing to evaluate how Vision-Language-Action (VLA) models generalize across diverse tasks. By generating over 28,000 trajectories from natural language prompts, the authors show that while current models can learn complex spatial compositions and multimodal behaviors, they still struggle with out-of-distribution object concepts and specific execution failures like "overdrawing."
What Problem Does This Paper Address?
Training Vision-Language-Action (VLA) models—the "brains" behind modern generalist robots—requires massive amounts of diverse data. However, collecting high-quality robot trajectories in the real world is prohibitively expensive and slow. Furthermore, existing benchmarks often allow for "visual shortcuts": if a robot is told to "pick up the red mug," it can simply locate the red pixels in its view without truly understanding the underlying concept.
The authors propose robotic drawing as a solution. Drawing starts with a blank canvas, forcing the model to generate a plan based purely on language instructions rather than reacting to pre-existing objects. It is also inherently multimodal—there are infinite ways to draw a "cat"—making it a perfect stress test for a model's ability to handle diversity and long-horizon planning.
Methods
The core of the paper is the BURGER pipeline, a scalable method for generating robot data:
- Prompt-to-Image: Uses LLMs (like GPT-Image 2) to generate diverse sketch candidates from text prompts.
- Image-to-Trajectory: Converts these PNG sketches into skeletons, then SVG strokes, and finally into robot end-effector trajectories.
- Cross-Simulator Execution: These trajectories are replayed in simulators like ManiSkill, MuJoCo, and Isaac Sim to collect RGB observations and proprioceptive data.
To evaluate the results, the authors developed a custom DINOv3-based metric. Unlike standard metrics like CLIPScore, which can be too "coarse," this evaluator is trained specifically on the drawing distribution to detect fine-grained semantic and spatial errors.
Datasets
The authors introduced the BURGER Dataset, which includes:
- 132 Single-Object Tasks: Ranging from animals to kitchenware.
- 150 Compositional Tasks: Testing spatial relations like "draw A to the left of B" or "draw A under B."
- 28,200 Total Simulated Demonstrations: 100 trajectories per task.
- Real-Robot Data: 100 demonstrations collected on a Franka Panda arm for sim-to-real validation.
Compared to existing benchmarks like LIBERO, BURGER offers a 6.6x increase in trajectory diversity for the same instruction, providing a much richer training signal.
Key Insights and Ideas
- Multimodality is Key: Drawing is a "one-to-many" problem. The researchers found that flow-matching models (like $\pi_0$) are significantly better at preserving this diversity than autoregressive models, which tend to collapse into a single "average" way of drawing.
- Zero-Shot Spatial Reasoning: One of the most exciting findings is that VLA models can generalize to unseen compositions. A model trained to draw "apples" and "cups" separately can successfully follow an instruction to "draw a cup to the left of an apple" even if it never saw that specific pair during training.
- The "Blank Canvas" Advantage: Because the robot starts with nothing, researchers can directly observe failure modes. For example, some models exhibit "failure to terminate," where they keep drawing over the same spot because they don't recognize the task is finished.
Key Results
- Scaling Laws: Performance generally improves as the number of training classes and demonstrations increases. Flow-matching models ($\pi_0$ and $\pi_{0.5}$) showed the most consistent benefits from scaling.
- Semantic Limits: While models generalize well to new spatial arrangements, they fail on out-of-distribution objects. If a model wasn't trained on "rocket ships," it produces unstructured scribbles when asked to draw one.
- Real-World Deployment: The authors successfully fine-tuned a $\pi_{0.5}$ policy on just 100 real-robot demonstrations. The robot was even able to react to human intervention—for instance, if a human drew a wheel on a bicycle, the robot would recognize it and move on to draw the rest of the frame.
Limitations and Future Work
- Simplified Physics: BURGER focuses on "sketch-level" drawing with a marker. It doesn't yet account for the complex fluid dynamics and contact-rich physics of brush painting or oil colors.
- Heuristic Biases: The automated pipeline uses specific heuristics for stroke order (e.g., drawing the outline first). This might introduce biases that don't perfectly reflect how a human would naturally draw.
- Future Directions: The authors suggest expanding the benchmark to include more complex artistic styles and exploring how these drawing-learned representations might transfer to more traditional manipulation tasks like assembly or sorting.
Summary updated