TL;DR
Current Vision-Language Models (VLMs) can write crochet patterns that look perfect on paper but produce "monsters" when actually executed. This paper introduces FoundationChain, a testbed that moves beyond text-based metrics to real-world physical execution, revealing that even top-tier models struggle with basic geometric and procedural reasoning.
What Problem Does This Paper Address?
We are increasingly asking AI to guide actions in the physical world—from assembling LEGO sets to assisting in chemistry labs. However, evaluating "procedural reasoning" (the ability to plan and execute a sequence of interdependent steps) is difficult. Most researchers rely on automated proxies like text similarity or simulations, which are often "blind" to functional failures.
The authors propose crochet as the ultimate "stress test" for VLMs. Unlike knitting, crochet is non-lattice-based and strictly sequential; a single error early on propagates through the entire object. It cannot be reliably simulated, yet it is cheap and tangible for a human to verify. The core question is: can a VLM look at an image of a simple crochet object and generate a pattern that actually reproduces it?
Methods
The researchers developed a three-stage "scaffold" to help models succeed while isolating where they fail:
- Description: The VLM identifies the object and its features.
- Planning: The VLM answers 11 specific questions about construction (e.g., "Is it worked in rows or rounds?", "How many stitches in the first ring?").
- Pattern Writing: The VLM generates the final step-by-step instructions.
Crucially, the authors then hand-crocheted all 40 generated patterns (representing 60 hours of manual labor) to see what they actually produced. They scored the results against a taxonomy of eight error types, including geometry, state tracking, and "hallucination" (adding features not present in the target).
Datasets
- FoundationChain: A new benchmark consisting of 10 elementary crochet targets (circles, squares, stars, etc.) designed to probe core aspects of pattern writing.
- CrochetBench: The authors also performed a meta-analysis of this existing benchmark, demonstrating that its automated metrics (like BLEU or ROUGE scores) fail to distinguish between a working pattern and one where every number has been replaced with a random value.
Key Insights and Ideas
- The "Convincing Paper" Trap: Models are excellent at mimicking the style of a pattern. They use the right abbreviations and "boilerplate" phrases, which tricks automated text metrics into giving them high scores. However, the underlying logic is often broken.
- Geometric Blindness: Models struggle with the relationship between stitch counts and curvature. For example, to keep a circle flat, you must add a specific number of stitches per round. Models often failed this "Gaussian curvature" test, producing cups or ruffles instead of flat discs.
- The "Turning" Problem: In crochet, you often turn the work over to start a new row, which reverses the order of stitches. Models frequently failed to account for this basic spatial transformation, leading to asymmetrical or shrinking objects.
- State Tracking Failures: Models often lose track of the "live loop." They might write a summary of a row (e.g., "12 stitches total") but then generate instructions for the next row that assume a completely different starting state.
Key Results
The results were a wake-up call for VLM capabilities:
- Only 4 out of 40 patterns were error-free. All four successful patterns came from Gemini 3.7 Flash.
- 82.5% of outputs were the wrong size. Models consistently failed at the "simple" task of counting rows or rounds in an image.
- Average of 3.53 errors per pattern. Most errors were not minor; they radically changed the resulting object.
- Gemini 3.7 Flash significantly outperformed GPT-5.6 Terra, Kimi K3, and Inkling, primarily due to superior visual perception and more accurate initial descriptions.
Limitations and Future Work
The study was small-scale (10 targets, 4 models) due to the high cost of manual execution. The authors also noted that all patterns were executed by a single expert who knew the ground truth, which could introduce bias, though they argue most failures were too catastrophic to be "rescued" by a favorable interpretation.
Future work aims to expand the target library and involve a larger group of independent crocheters for "blind" scoring. The authors suggest that crochet remains a promising, low-cost domain for testing the "frontier" of physically grounded AI reasoning.
Summary updated