Evaluating Physically Grounded Procedural Reasoning in VLMs through Real-World Execution: Crochet as a Testbed

Maximiliano WardlearXiv 2026 · arXiv

TL;DR

Current Vision-Language Models (VLMs) can write crochet patterns that look perfect on paper but produce "monsters" when actually executed. This paper introduces FoundationChain, a testbed that moves beyond text-based metrics to real-world physical execution, revealing that even top-tier models struggle with basic geometric and procedural reasoning.

What Problem Does This Paper Address?

We are increasingly asking AI to guide actions in the physical world—from assembling LEGO sets to assisting in chemistry labs. However, evaluating "procedural reasoning" (the ability to plan and execute a sequence of interdependent steps) is difficult. Most researchers rely on automated proxies like text similarity or simulations, which are often "blind" to functional failures.

The authors propose crochet as the ultimate "stress test" for VLMs. Unlike knitting, crochet is non-lattice-based and strictly sequential; a single error early on propagates through the entire object. It cannot be reliably simulated, yet it is cheap and tangible for a human to verify. The core question is: can a VLM look at an image of a simple crochet object and generate a pattern that actually reproduces it?

Figure 2: A–D: making a ‘double crochet’ stitch. Yarn is wrapped over the hook (A), the hook is inserted into the target stitch and yarn is drawn back through (B), then through two of the loops on the hook (C), and finally through the last two loops (D), completing the stitch and returning to the state in A. E: making a ‘foundation chain’, which acts as the base for the first row of stitches in a pattern. F: working ‘in the round’ to make a circle.

Methods

The researchers developed a three-stage "scaffold" to help models succeed while isolating where they fail:

  1. Description: The VLM identifies the object and its features.
  2. Planning: The VLM answers 11 specific questions about construction (e.g., "Is it worked in rows or rounds?", "How many stitches in the first ring?").
  3. Pattern Writing: The VLM generates the final step-by-step instructions.

Crucially, the authors then hand-crocheted all 40 generated patterns (representing 60 hours of manual labor) to see what they actually produced. They scored the results against a taxonomy of eight error types, including geometry, state tracking, and "hallucination" (adding features not present in the target).

Datasets

  • FoundationChain: A new benchmark consisting of 10 elementary crochet targets (circles, squares, stars, etc.) designed to probe core aspects of pattern writing.
  • CrochetBench: The authors also performed a meta-analysis of this existing benchmark, demonstrating that its automated metrics (like BLEU or ROUGE scores) fail to distinguish between a working pattern and one where every number has been replaced with a random value.

Key Insights and Ideas

  • The "Convincing Paper" Trap: Models are excellent at mimicking the style of a pattern. They use the right abbreviations and "boilerplate" phrases, which tricks automated text metrics into giving them high scores. However, the underlying logic is often broken.
  • Geometric Blindness: Models struggle with the relationship between stitch counts and curvature. For example, to keep a circle flat, you must add a specific number of stitches per round. Models often failed this "Gaussian curvature" test, producing cups or ruffles instead of flat discs.
  • The "Turning" Problem: In crochet, you often turn the work over to start a new row, which reverses the order of stitches. Models frequently failed to account for this basic spatial transformation, leading to asymmetrical or shrinking objects.
  • State Tracking Failures: Models often lose track of the "live loop." They might write a summary of a row (e.g., "12 stitches total") but then generate instructions for the next row that assume a completely different starting state.
Stitch counts per round for the Circle target. Models often state a goal (flat circle) but generate instructions that do the opposite.

Key Results

The results were a wake-up call for VLM capabilities:

  • Only 4 out of 40 patterns were error-free. All four successful patterns came from Gemini 3.7 Flash.
  • 82.5% of outputs were the wrong size. Models consistently failed at the "simple" task of counting rows or rounds in an image.
  • Average of 3.53 errors per pattern. Most errors were not minor; they radically changed the resulting object.
  • Gemini 3.7 Flash significantly outperformed GPT-5.6 Terra, Kimi K3, and Inkling, primarily due to superior visual perception and more accurate initial descriptions.
Heatmap showing error categories across each model's output patterns. Success (the checkmark) was rare.

Limitations and Future Work

The study was small-scale (10 targets, 4 models) due to the high cost of manual execution. The authors also noted that all patterns were executed by a single expert who knew the ground truth, which could introduce bias, though they argue most failures were too catastrophic to be "rescued" by a favorable interpretation.

Future work aims to expand the target library and involve a larger group of independent crocheters for "blind" scoring. The authors suggest that crochet remains a promising, low-cost domain for testing the "frontier" of physically grounded AI reasoning.

Summary updated

Abstract

Vision-Language Models (VLMs) can generate procedures that look convincing on paper, but do they work in practice? In this paper, we propose crochet as an ideal testbed for physically grounded procedural reasoning in VLMs. Crochet cannot be reliably simulated or automated: the only way to know what a pattern produces is to make it by hand, yet verification is tangible and cheap, requiring nothing but a hook and a ball of yarn. We introduce FoundationChain, a deceptively simple test of procedural reasoning: given an image of a basic crochet object, can VLMs infer how it was made and generate a pattern that produces the same result? We crochet 40 patterns generated by four VLMs by hand. Even at this scale, physical execution not only reveals failures that automated proxies cannot detect, but also enables a more precise diagnosis of their causes and consequences: while models write fluent, convincing patterns, only four are free of errors, and our analysis exposes deficiencies in visual, geometric and procedural reasoning.
BibTeX
@misc{wardle2026evaluating,
  author = {Maximiliano Wardle},
  title = {{Evaluating Physically Grounded Procedural Reasoning in VLMs through Real-World Execution: Crochet as a Testbed}},
  year = {2026},
  eprint = {2611.10740},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10740},
}
Figure