Abstract
Large Language Models (LLMs) often generate fluent but physically impossible instructions for complex assembly tasks like sewing. This paper introduces a novel tree-based evaluation metric designed to measure spatiotemporal consistency by extracting the underlying assembly logic into a hierarchical structure. Experimental results demonstrate that this metric is significantly more effective than traditional text-similarity measures at detecting logical errors and out-of-order steps.
Introduction
When an LLM generates a cooking recipe or a sewing tutorial, the text might look professional while being physically nonsensical. In sewing, the order of operations is critical: one cannot attach a pocket to a skirt that hasn't been assembled, nor sew a sleeve to a bodice if the armhole does not yet exist.
Traditional evaluation metrics like BLEU, ROUGE, and BERTScore focus on surface-level similarity—measuring how much the generated text resembles a human-written reference. However, these metrics are notoriously blind to spatiotemporal consistency: the understanding of where objects are (spatial) and how they change over time (temporal). A model could shuffle the steps of a complex pattern, and traditional metrics would still provide a high score because the vocabulary remains unchanged. This research addresses the need for a metric that evaluates whether instructions are physically executable.
The Tree-Based Evaluation Metric
The authors propose a tree-based evaluation metric that represents a garment's assembly as a hierarchical structure. This approach moves beyond text matching to capture the logic of physical construction.
Methodology
- Gold Tree Annotation: Experts define "gold trees" using Context-Free Grammar (CFG) rules. These trees represent all valid sequences for joining pieces (e.g., Piece A + Piece B = Component AB).
- Tree Extraction: To evaluate an LLM's output, a secondary "Piece Extraction LLM" identifies which fabric pieces are mentioned in each step.
- Subtree Construction: Extracted pieces are converted into subtrees. For example, a step describing attaching a pocket (A) to a front panel (B) creates a node
ABwith childrenAandB. - Self-Attachment Handling: The metric accounts for "self-attachment" (e.g., sewing a flat piece into a tube) using integer-valued counters on nodes.
- Scoring: The final "Tree Score" is the maximum $F_1$ score achieved by comparing the automatically extracted tree against the set of valid gold trees.
Dataset and Complexity
The study utilizes a curated dataset based on patterns from moodfabrics.com, covering 22 patterns across five categories: shirts, dresses, pants, skirts, and jumpsuits. Each pattern includes an overview image and descriptions of pieces labeled A-Z.
Because garments can often be assembled in multiple valid orders, the dataset includes an average of 1,709.0 possible gold trees per pattern, generated via CFG rules. This complexity highlights the difficulty LLMs face in maintaining logical consistency across many valid permutations.
Key Results
Sensitivity to Permutation
In a "shuffling" experiment, researchers randomly reordered the steps of valid instruction sets. While BLEU and BERTScore scores remained nearly identical (changing by only -0.001 and -0.003 respectively), the Tree Score dropped by approximately 47% (-0.240). This demonstrates that traditional metrics essentially "approve" nonsensical, out-of-order instructions, whereas the Tree Score correctly identifies the loss of logic.
Correlation with Human Judgment
The Tree Score achieved a Pearson correlation of -0.599 with manually annotated error counts. In contrast, BLEU (-0.111) and BERTScore (-0.080) showed no statistically significant correlation with actual errors. Furthermore, the Tree Score was the only metric that positively correlated with human ratings of step-to-step logic ($r = 0.23, p < .05$), while BLEU actually showed a negative correlation, suggesting it may reward wordiness over coherence.
Discussion and Future Work
While the Tree Score represents a significant advancement, it currently tracks which pieces are attached but not which specific edges (seams) are being joined. It identifies that Piece A is attached to Piece B, but not whether the attachment occurs at the shoulder or the waist.
Additionally, because the trees are built bottom-up, a single error in an early step can cause subsequent tree construction to fail, potentially over-penalizing a model for a single mislabeled piece. Future work aims to adapt this CFG-based evaluation for other assembly tasks, such as furniture construction or Minecraft building, where logical sequencing is equally paramount.
Summary updated