BURGER: Benchmarking and Understanding Robotic Generalization via Embodied Drawing

Xiangyu Xu, Bangshuai Peng, Ziyi Liu, Mengjie Zhao, Chenhao Lu, Zhengyang Fan, Huazhe Xu, Kaizhe HuarXiv 2026 · arXiv

TL;DR

BURGER is a new benchmark that uses robotic drawing to evaluate how Vision-Language-Action (VLA) models generalize across diverse tasks. By generating over 28,000 trajectories from natural language prompts, the authors show that while current models can learn complex spatial compositions and multimodal behaviors, they still struggle with out-of-distribution object concepts and specific execution failures like "overdrawing."

What Problem Does This Paper Address?

Training Vision-Language-Action (VLA) models—the "brains" behind modern generalist robots—requires massive amounts of diverse data. However, collecting high-quality robot trajectories in the real world is prohibitively expensive and slow. Furthermore, existing benchmarks often allow for "visual shortcuts": if a robot is told to "pick up the red mug," it can simply locate the red pixels in its view without truly understanding the underlying concept.

The authors propose robotic drawing as a solution. Drawing starts with a blank canvas, forcing the model to generate a plan based purely on language instructions rather than reacting to pre-existing objects. It is also inherently multimodal—there are infinite ways to draw a "cat"—making it a perfect stress test for a model's ability to handle diversity and long-horizon planning.

Methods

The core of the paper is the BURGER pipeline, a scalable method for generating robot data:

  1. Prompt-to-Image: Uses LLMs (like GPT-Image 2) to generate diverse sketch candidates from text prompts.
  2. Image-to-Trajectory: Converts these PNG sketches into skeletons, then SVG strokes, and finally into robot end-effector trajectories.
  3. Cross-Simulator Execution: These trajectories are replayed in simulators like ManiSkill, MuJoCo, and Isaac Sim to collect RGB observations and proprioceptive data.
Figure 1: Overview of BURGER. Top: Prompt-generated PNG images are converted into skeletons, SVG strokes, and robot trajectories. Bottom left: BURGER tests semantic grounding, spatial rea- soning, and multimodal action generation in VLA models. Bottom right: We evaluate models on single-object drawing, compositional drawing, data scaling, and real-world deployment.

To evaluate the results, the authors developed a custom DINOv3-based metric. Unlike standard metrics like CLIPScore, which can be too "coarse," this evaluator is trained specifically on the drawing distribution to detect fine-grained semantic and spatial errors.

Datasets

The authors introduced the BURGER Dataset, which includes:

  • 132 Single-Object Tasks: Ranging from animals to kitchenware.
  • 150 Compositional Tasks: Testing spatial relations like "draw A to the left of B" or "draw A under B."
  • 28,200 Total Simulated Demonstrations: 100 trajectories per task.
  • Real-Robot Data: 100 demonstrations collected on a Franka Panda arm for sim-to-real validation.

Compared to existing benchmarks like LIBERO, BURGER offers a 6.6x increase in trajectory diversity for the same instruction, providing a much richer training signal.

Key Insights and Ideas

  • Multimodality is Key: Drawing is a "one-to-many" problem. The researchers found that flow-matching models (like $\pi_0$) are significantly better at preserving this diversity than autoregressive models, which tend to collapse into a single "average" way of drawing.
  • Zero-Shot Spatial Reasoning: One of the most exciting findings is that VLA models can generalize to unseen compositions. A model trained to draw "apples" and "cups" separately can successfully follow an instruction to "draw a cup to the left of an apple" even if it never saw that specific pair during training.
  • The "Blank Canvas" Advantage: Because the robot starts with nothing, researchers can directly observe failure modes. For example, some models exhibit "failure to terminate," where they keep drawing over the same spot because they don't recognize the task is finished.
Figure 2: Instruction-conditioned planar trajectory diversity. The three rows show LIBERO demon- strations, BURGER demonstrations, and π0 rollouts, respectively. Each subplot overlays trajectories for the same instruction. Different modes correspond to distinct clusters of execution routes.

Key Results

  • Scaling Laws: Performance generally improves as the number of training classes and demonstrations increases. Flow-matching models ($\pi_0$ and $\pi_{0.5}$) showed the most consistent benefits from scaling.
  • Semantic Limits: While models generalize well to new spatial arrangements, they fail on out-of-distribution objects. If a model wasn't trained on "rocket ships," it produces unstructured scribbles when asked to draw one.
  • Real-World Deployment: The authors successfully fine-tuned a $\pi_{0.5}$ policy on just 100 real-robot demonstrations. The robot was even able to react to human intervention—for instance, if a human drew a wheel on a bicycle, the robot would recognize it and move on to draw the rest of the frame.
The real-world hardware setup using a Franka Panda arm and dual RealSense cameras.

Limitations and Future Work

  • Simplified Physics: BURGER focuses on "sketch-level" drawing with a marker. It doesn't yet account for the complex fluid dynamics and contact-rich physics of brush painting or oil colors.
  • Heuristic Biases: The automated pipeline uses specific heuristics for stroke order (e.g., drawing the outline first). This might introduce biases that don't perfectly reflect how a human would naturally draw.
  • Future Directions: The authors suggest expanding the benchmark to include more complex artistic styles and exploring how these drawing-learned representations might transfer to more traditional manipulation tasks like assembly or sorting.

Summary updated

Abstract

How effectively can vision-language-action (VLA) models learn diverse language-conditioned behaviors as training data scales? Investigating this question requires large and varied robot datasets, whose collection on physical robots remains costly. We introduce BURGER, a language-conditioned robotic drawing benchmark with a scalable, prompt-driven pipeline for generating continuous drawing trajectories, both in simulation and the real world. In BURGER, an agent starts from the same blank canvas and controls a robot arm to draw the object or spatial composition specified by language, while each instruction allows an open-ended variety of valid drawings and stroke orders. Using 28,200 simulated demonstrations spanning 282 drawing tasks, we evaluate five representative VLA models across data scales, language grounding, and multimodal behavior. BURGER supports fine-grained analysis of language grounding, spatial composition, and action generation, revealing distinct failure patterns across VLA models. Our experiments suggest that some VLA models can benefit from greater task diversity and demonstration scale, capture multimodal drawing behaviors, and generalize to unseen compositions of familiar objects. Finally, we deploy a trained policy on a Franka Panda arm, demonstrating the ability to draw reactively under human intervention. Project homepage and code can be found at https://siyingzhao.github.io/BURGER/.
BibTeX
@misc{xu2026burger,
  author = {Xiangyu Xu and Bangshuai Peng and Ziyi Liu and Mengjie Zhao and Chenhao Lu and Zhengyang Fan and Huazhe Xu and Kaizhe Hu},
  title = {{BURGER: Benchmarking and Understanding Robotic Generalization via Embodied Drawing}},
  year = {2026},
  eprint = {2610.40532},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.40532},
}
Figure