Evaluating Spatiotemporal Consistency in Automatically Generated Sewing Instructions

Luisa Geiger, Mareike Hartmann, Michael Sullivan, Alexander Koller

Abstract

Large Language Models (LLMs) often generate fluent but physically impossible instructions for complex assembly tasks like sewing. This paper introduces a novel tree-based evaluation metric designed to measure spatiotemporal consistency by extracting the underlying assembly logic into a hierarchical structure. Experimental results demonstrate that this metric is significantly more effective than traditional text-similarity measures at detecting logical errors and out-of-order steps.

Introduction

When an LLM generates a cooking recipe or a sewing tutorial, the text might look professional while being physically nonsensical. In sewing, the order of operations is critical: one cannot attach a pocket to a skirt that hasn't been assembled, nor sew a sleeve to a bodice if the armhole does not yet exist.

Traditional evaluation metrics like BLEU, ROUGE, and BERTScore focus on surface-level similarity—measuring how much the generated text resembles a human-written reference. However, these metrics are notoriously blind to spatiotemporal consistency: the understanding of where objects are (spatial) and how they change over time (temporal). A model could shuffle the steps of a complex pattern, and traditional metrics would still provide a high score because the vocabulary remains unchanged. This research addresses the need for a metric that evaluates whether instructions are physically executable.

The Tree-Based Evaluation Metric

The authors propose a tree-based evaluation metric that represents a garment's assembly as a hierarchical structure. This approach moves beyond text matching to capture the logic of physical construction.

Extraction of a hierarchical assembly tree from text-based sewing instructions.

Methodology

  1. Gold Tree Annotation: Experts define "gold trees" using Context-Free Grammar (CFG) rules. These trees represent all valid sequences for joining pieces (e.g., Piece A + Piece B = Component AB).
  2. Tree Extraction: To evaluate an LLM's output, a secondary "Piece Extraction LLM" identifies which fabric pieces are mentioned in each step.
  3. Subtree Construction: Extracted pieces are converted into subtrees. For example, a step describing attaching a pocket (A) to a front panel (B) creates a node AB with children A and B.
  4. Self-Attachment Handling: The metric accounts for "self-attachment" (e.g., sewing a flat piece into a tube) using integer-valued counters on nodes.
  5. Scoring: The final "Tree Score" is the maximum $F_1$ score achieved by comparing the automatically extracted tree against the set of valid gold trees.

Dataset and Complexity

The study utilizes a curated dataset based on patterns from moodfabrics.com, covering 22 patterns across five categories: shirts, dresses, pants, skirts, and jumpsuits. Each pattern includes an overview image and descriptions of pieces labeled A-Z.

Mapping of 2D pattern pieces to 3D garment seams on a mannequin.

Because garments can often be assembled in multiple valid orders, the dataset includes an average of 1,709.0 possible gold trees per pattern, generated via CFG rules. This complexity highlights the difficulty LLMs face in maintaining logical consistency across many valid permutations.

Key Results

Sensitivity to Permutation

In a "shuffling" experiment, researchers randomly reordered the steps of valid instruction sets. While BLEU and BERTScore scores remained nearly identical (changing by only -0.001 and -0.003 respectively), the Tree Score dropped by approximately 47% (-0.240). This demonstrates that traditional metrics essentially "approve" nonsensical, out-of-order instructions, whereas the Tree Score correctly identifies the loss of logic.

Correlation with Human Judgment

The Tree Score achieved a Pearson correlation of -0.599 with manually annotated error counts. In contrast, BLEU (-0.111) and BERTScore (-0.080) showed no statistically significant correlation with actual errors. Furthermore, the Tree Score was the only metric that positively correlated with human ratings of step-to-step logic ($r = 0.23, p < .05$), while BLEU actually showed a negative correlation, suggesting it may reward wordiness over coherence.

Discussion and Future Work

While the Tree Score represents a significant advancement, it currently tracks which pieces are attached but not which specific edges (seams) are being joined. It identifies that Piece A is attached to Piece B, but not whether the attachment occurs at the shoulder or the waist.

Additionally, because the trees are built bottom-up, a single error in an early step can cause subsequent tree construction to fail, potentially over-penalizing a model for a single mislabeled piece. Future work aims to adapt this CFG-based evaluation for other assembly tasks, such as furniture construction or Minecraft building, where logical sequencing is equally paramount.

Summary updated

Abstract

In this paper, we propose a novel, automatic tree-based evaluation metric for LLM-generated step-by-step assembly instructions, that more accurately reflects spatiotemporal aspects of construction than traditional metrics such as BLEU and BERT similarity scores. We apply our proposed metric to the domain of sewing instructions, and show that our metric better correlates with manually-annotated error counts as well as human quality ratings, demonstrating our metric's superiority for evaluating the spatiotemporal soundness of sewing instructions. Further experiments show that our metric is more robust than traditional approaches against artificially-constructed counterfactual examples that are specifically constructed to confound metrics that rely on textual similarity.
BibTeX
@misc{geiger2025evaluating,
  author = {Luisa Geiger and Mareike Hartmann and Michael Sullivan and Alexander Koller},
  title = {{Evaluating Spatiotemporal Consistency in Automatically Generated Sewing Instructions}},
  year = {2025},
  eprint = {2509.24792},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.24792},
}
Figure