Structured Document Generation for Industrial Equipment

Karol Lynch, Fabio Lorenzi, John D. Sheehan, Duygu Kabakci-Zorlu, Bradley EckAAAI 2025 · DOI

TL;DR

This paper introduces a novel AI system that leverages Large Language Models (LLMs) to automatically generate complex, structured technical documents like Failure Modes and Effects Analysis (FMEAs) for industrial equipment. By combining in-context learning, structured generation techniques, and a unique ensemble method, the system significantly speeds up the creation of these crucial documents while maintaining high quality and allowing for expert supervision.

What Problem Does This Paper Address?

Creating Failure Modes and Effects Analysis (FMEAs) for industrial equipment is a critical but labor-intensive process. FMEAs document potential failure points, their causes, and consequences, and are essential for safety and maintenance. Traditionally, this involves a team of experts and can take days or even weeks to complete, making it prohibitively expensive for many companies to generate FMEAs for all their equipment. Furthermore, FMEAs are highly structured, domain-specific documents with complex hierarchical relationships, posing a significant challenge for current AI methods that often struggle with maintaining structural integrity and semantic accuracy in generated content. This paper aims to bridge this gap by developing an AI-powered system that can generate high-quality, structured FMEAs efficiently, reducing the manual effort required.

Methods

The authors propose a multi-faceted approach to structured document generation:

  • In-Context Learning (ICL) with User Supervision (SDFSP): They adapt existing ICL techniques by incorporating human oversight into the example selection process. This "Supervised Dynamic Few-Shot Prompting" (SDFSP) involves retrieving relevant examples from a large database of existing FMEAs and then allowing users to refine the selection, ensuring the LLM is guided by both data and expert knowledge.
  • Structured Generation: To ensure the generated FMEAs adhere to their inherent hierarchical structure, the system uses a dynamic prompting strategy that issues content-dependent prompts for each FMEA step. This is complemented by robust rule-based parsers to transform raw LLM text into structured data formats (like lists) and data validation using Pydantic to enforce structural integrity. This approach also helps mitigate repetition issues common in LLM outputs.
  • Counter-Based Entity Ensembling (CBEE): A novel ensemble method is introduced to merge outputs from multiple LLMs or prompts. CBEE leverages the semantic similarity of entities within generated content to create a consensus, leading to more coherent and accurate final documents. This method is flexible and can be tuned to balance recall and precision.
  • Structured Semantic Entity Evaluation (SSEE) Metrics: To address the limitations of traditional metrics (like ROUGE, BLEU) that don't account for document structure or semantic meaning, the authors propose SSEE metrics. These metrics evaluate both the structure and semantic accuracy of generated content, aligning with user concerns about specific document elements.

Datasets

The primary dataset used in this research is a proprietary database of FMEAs developed by Subject Matter Experts (SMEs). This database contains 714 examples, which were split into training (80%), validation (10%), and test (10%) sets. Examples for ICL were drawn from the training split.

Key Insights and Ideas

  • Human-in-the-loop is Crucial for Structured Data: While LLMs are powerful, directly applying them to highly structured, domain-specific documents like FMEAs requires human supervision, especially in example selection, to achieve high quality.
  • Structure Matters: Traditional text generation metrics are insufficient for evaluating structured documents. The development of SSEE metrics highlights the importance of evaluating both the semantic content and the hierarchical organization of generated text.
  • Ensembling for Robustness: Merging outputs from multiple LLMs using a semantic-aware ensemble method (CBEE) significantly improves the reliability and accuracy of generated structured content, outperforming individual models.
  • Efficiency Gains are Substantial: Even with expert supervision, the AI-assisted generation process represents a significant time saving compared to traditional manual FMEA creation, moving from "several person days" to "several person hours."

Key Results

  • Individual Model Performance: Llama 3 emerged as the best-performing individual model, achieving an SSEE Recall of 0.605 and SSEE F1 of 0.587. Including more shots (15 vs. 5) generally improved results, though SSEE Precision saw a slight dip, possibly due to irrelevant content in the extra shots.
  • Ensemble Performance: The CBEE ensemble approach demonstrated significant improvements. With a count threshold of 1, recall increased by over 10% (to 0.669). A count threshold of 5 achieved a remarkable precision of 0.91. The ensemble with a threshold of 2 provided the best balance for ROUGE-1 and SSEE F1.
  • Overall Effectiveness: The system achieved an overall effectiveness with a recall between 0.669 and a precision of 0.91, demonstrating its practical utility.
  • User Feedback: SMEs responded overwhelmingly positively to the application, appreciating the balance between automation and supervision. An expert engineer found the AI-assisted process to be a significant improvement over traditional methods.

Limitations and Future Work

  • Latency: User feedback highlighted the importance of low-latency responses for a smooth interactive experience.
  • Deployment Complexity: Integrating with diverse enterprise asset management solutions and standardizing APIs across various LLM providers and vector databases presents ongoing challenges.
  • Future Work: The authors plan to focus on finalizing deployment to end-users. They also intend to explore alternative methods for enhancing content quality, such as synthetic data generation and task-specific fine-tuning of LLMs. Investigating different ensemble strategies and further refining the SSEE metrics are also potential avenues.

Summary updated

BibTeX
@inproceedings{lynch2025structured,
  author = {Karol Lynch and Fabio Lorenzi and John D. Sheehan and Duygu Kabakci-Zorlu and Bradley Eck},
  title = {{Structured Document Generation for Industrial Equipment}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35150},
}
Figure