Abstract: In this paper, we propose a novel, automatic tree-based evaluation metric for LLM-generated step-by-step assembly instructions, that more accurately reflects spatiotemporal aspects of construction than traditional metrics such as BLEU and BERT similarity scores. We apply our proposed metric to the domain of sewing instructions, and show that our metric better correlates with manually-annotated error counts as well as human quality ratings, demonstrating our metric's superiority for evaluating the spatiotemporal soundness of sewing instructions. Further experiments show that our metric is more robust than traditional approaches against artificially-constructed counterfactual examples that are specifically constructed to confound metrics that rely on textual similarity.
Introduction
Large Language Models (LLMs) have demonstrated impressive capabilities in generating descriptive text, but they often struggle with the physical logic required for assembly instructions. In domains like sewing, a set of instructions is not merely a sequence of words but a description of physical transformations. A single logical error—such as instructing a user to sew a sleeve shut before attaching it to the bodice—renders the entire tutorial useless.
Standard NLP evaluation tools like BLEU, ROUGE, and BERTScore are fundamentally "blind" to this physical logic. They measure textual overlap or semantic similarity rather than the validity of the underlying "world state." This paper addresses the need for metrics that understand which components are connected and in what order.
Methodology: Tree-Based Evaluation
The core contribution of this work is a Tree-Based Evaluation Metric designed to capture the hierarchical nature of garment assembly. The methodology consists of three primary stages:
- Gold Tree Annotation: Experts define "gold trees" using Context-Free Grammars (CFGs) for each pattern. These trees represent all valid assembly paths (e.g., "Piece A + Piece B = Component AB").
- Automatic Tree Extraction: An LLM parses the generated instructions to identify which pieces are joined at each step. These actions are converted into a "predicted" assembly tree.
- Tree Comparison: The predicted tree is compared against the gold trees. The final "Tree Score" is the maximum $F_1$ score achieved against any valid gold assembly path.
To handle "self-attachment" (e.g., sewing a flat piece into a tube), the metric utilizes integer-valued counters on tree nodes to track component modifications.
Experimental Setup and Datasets
The study utilized a newly curated dataset based on 22 distinct sewing patterns from moodfabrics.com, spanning categories like shirts, dresses, and pants. The researchers generated 176 instruction sets using GPT-4o with various prompting strategies. To establish a ground truth, 110 instruction sets were manually rated by 20 participants with sewing experience.
Key Results and Insights
The Failure of Text Similarity
The experiments revealed that traditional metrics are poor proxies for physical logic. BERTScore showed almost zero correlation with the actual number of physical assembly errors. To further test robustness, the authors created "shuffled" counterfactuals by randomly permuting instruction steps. While BERTScore dropped by less than 1%, the Tree Score plummeted by nearly 50%, correctly identifying the instructions as nonsensical.
Correlation with Human Judgment
The Tree Score demonstrated a strong negative correlation (-0.599) with manually annotated error counts, whereas BLEU (-0.111) and BERTScore (-0.080) showed no significant correlation. Furthermore, the Tree Score was the only metric that aligned with human ratings regarding step-by-step coherence.
Limitations and Future Work
While the tree-based metric significantly outperforms traditional measures, it currently lacks granularity regarding specific edges. It tracks which pieces are joined but not where (e.g., side seam vs. shoulder seam).
Future work aims to expand the tree logic to handle more complex garments with duplicate pieces (like ruffles) and to generalize the framework to other assembly domains such as furniture (IKEA) or LEGO construction.
Summary updated