Evaluating Spatiotemporal Consistency in Automatically Generated Sewing Instructions

Luisa Geiger, Mareike Hartmann, Michael Sullivan, Alexander Koller

Abstract: In this paper, we propose a novel, automatic tree-based evaluation metric for LLM-generated step-by-step assembly instructions, that more accurately reflects spatiotemporal aspects of construction than traditional metrics such as BLEU and BERT similarity scores. We apply our proposed metric to the domain of sewing instructions, and show that our metric better correlates with manually-annotated error counts as well as human quality ratings, demonstrating our metric's superiority for evaluating the spatiotemporal soundness of sewing instructions. Further experiments show that our metric is more robust than traditional approaches against artificially-constructed counterfactual examples that are specifically constructed to confound metrics that rely on textual similarity.

Introduction

Large Language Models (LLMs) have demonstrated impressive capabilities in generating descriptive text, but they often struggle with the physical logic required for assembly instructions. In domains like sewing, a set of instructions is not merely a sequence of words but a description of physical transformations. A single logical error—such as instructing a user to sew a sleeve shut before attaching it to the bodice—renders the entire tutorial useless.

Standard NLP evaluation tools like BLEU, ROUGE, and BERTScore are fundamentally "blind" to this physical logic. They measure textual overlap or semantic similarity rather than the validity of the underlying "world state." This paper addresses the need for metrics that understand which components are connected and in what order.

Methodology: Tree-Based Evaluation

The core contribution of this work is a Tree-Based Evaluation Metric designed to capture the hierarchical nature of garment assembly. The methodology consists of three primary stages:

  1. Gold Tree Annotation: Experts define "gold trees" using Context-Free Grammars (CFGs) for each pattern. These trees represent all valid assembly paths (e.g., "Piece A + Piece B = Component AB").
  2. Automatic Tree Extraction: An LLM parses the generated instructions to identify which pieces are joined at each step. These actions are converted into a "predicted" assembly tree.
  3. Tree Comparison: The predicted tree is compared against the gold trees. The final "Tree Score" is the maximum $F_1$ score achieved against any valid gold assembly path.
Figure 1: Example of sewing instructions mapped to a hierarchical assembly tree.

To handle "self-attachment" (e.g., sewing a flat piece into a tube), the metric utilizes integer-valued counters on tree nodes to track component modifications.

Experimental Setup and Datasets

The study utilized a newly curated dataset based on 22 distinct sewing patterns from moodfabrics.com, spanning categories like shirts, dresses, and pants. The researchers generated 176 instruction sets using GPT-4o with various prompting strategies. To establish a ground truth, 110 instruction sets were manually rated by 20 participants with sewing experience.

Key Results and Insights

The Failure of Text Similarity

The experiments revealed that traditional metrics are poor proxies for physical logic. BERTScore showed almost zero correlation with the actual number of physical assembly errors. To further test robustness, the authors created "shuffled" counterfactuals by randomly permuting instruction steps. While BERTScore dropped by less than 1%, the Tree Score plummeted by nearly 50%, correctly identifying the instructions as nonsensical.

Correlation with Human Judgment

The Tree Score demonstrated a strong negative correlation (-0.599) with manually annotated error counts, whereas BLEU (-0.111) and BERTScore (-0.080) showed no significant correlation. Furthermore, the Tree Score was the only metric that aligned with human ratings regarding step-by-step coherence.

Limitations and Future Work

While the tree-based metric significantly outperforms traditional measures, it currently lacks granularity regarding specific edges. It tracks which pieces are joined but not where (e.g., side seam vs. shoulder seam).

Figure 5: Visualization of specific seam types (e.g., princess seam, side seam) which current tree models do not yet distinguish.

Future work aims to expand the tree logic to handle more complex garments with duplicate pieces (like ruffles) and to generalize the framework to other assembly domains such as furniture (IKEA) or LEGO construction.

Summary updated

Abstract

In this paper, we propose a novel, automatic tree-based evaluation metric for LLMgenerated step-by-step assembly instructions, that more accurately reflects spatiotemporal aspects of construction than traditional metrics such as BLEU and BERT similarity scores.We apply our proposed metric to the domain of sewing instructions, and show that our metric better correlates with manually-annotated error counts as well as human quality ratings, demonstrating our metric's superiority for evaluating the spatiotemporal soundness of sewing instructions.Further experiments show that our metric is more robust than traditional approaches against artificially-constructed counterfactual examples that are specifically constructed to confound metrics that rely on textual similarity.
BibTeX
@misc{geiger2025evaluating,
  author = {Luisa Geiger and Mareike Hartmann and Michael Sullivan and Alexander Koller},
  title = {{Evaluating Spatiotemporal Consistency in Automatically Generated Sewing Instructions}},
  year = {2025},
  eprint = {2509.24792},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.24792},
}
Figure