Towards Safer Operations: An Expert-involved Dataset of High-Pressure Gas Incidents for Preventing Future Failures

Shumpei Inoue, Minh-Tien Nguyen, Hiroki Mizokuchi, Tuan-Anh D. Nguyen, Huu-Hiep Nguyen, Dung LeEMNLP 2023 · DOI

TL;DR

This paper introduces IncidentAI, a novel, expert-annotated Japanese dataset for high-pressure gas incidents, designed to advance AI-driven safety prevention. It addresses the critical need for high-quality data in this domain by providing annotations for three key NLP tasks: Named Entity Recognition (NER), Cause-Effect Extraction (CE), and Information Retrieval (IR).

What Problem Does This Paper Address?

Ensuring safety in high-pressure gas operations is paramount, as incidents can have severe consequences. While AI offers promising solutions for preventing future failures by analyzing historical incident reports, a significant gap exists due to the lack of high-quality, annotated datasets. Creating such datasets is challenging because it requires domain experts with deep knowledge and experience, and data collection is often hindered by company policies. This paper tackles this challenge by developing a specialized dataset for the high-pressure gas industry, enabling more effective AI applications for safety analysis and prevention.

Methods

The core of this work is the creation of the IncidentAI dataset, which focuses on three NLP tasks:

  • Named Entity Recognition (NER): The goal is to identify and classify key entities within incident reports. The authors established six entity types: Products, Chemicals, Storage, Incidents, Processes, and Tests. They developed detailed annotation guidelines and achieved a good inter-annotator agreement (Fleiss' Kappa of 0.814 for NER).
  • Cause-Effect Extraction (CE): This task aims to identify the causes of incidents and their resulting effects. The annotation process involved defining specific cause and effect types and achieved a Fleiss' Kappa of 0.764.
  • Information Retrieval (IR): The objective is to enable users to query incident descriptions and retrieve relevant past incidents. Instead of simple relevance scores, they developed a method based on evaluating relevance across a set of key attributes, achieving an average Fleiss' Kappa of 0.541 for IR annotation.

The paper also benchmarks several NLP models for each task, including layered nested NER, BERT-based models for CE, and bi-encoder models for IR.

Datasets

The primary contribution is the IncidentAI dataset, which is:

  • Domain-specific: Focuses on high-pressure gas incidents.
  • Multilingual: Primarily in Japanese, with English translations provided for some samples.
  • Multi-task: Annotated for NER, CE, and IR.
  • Expert-involved: Annotated by domain experts with at least six years of experience.

The dataset was initially collected from 18,171 incident reports published in 2022 by the High-Pressure Gas Safety Institute of Japan. After filtering for relevant industries ("general chemistry," "petrochemical," and "oil refining"), 2,159 cases were selected for IR annotation, and 970 cases were further selected for NER and CE annotation.

Key Insights and Ideas

  • Expert annotation is crucial: The significant effort in developing clear guidelines and achieving high inter-annotator agreement highlights the necessity of domain expert involvement for creating high-quality safety-critical datasets.
  • Nested NER is challenging but important: The paper demonstrates that entities can be nested (e.g., a product within a storage unit), and specialized models are needed to capture these complex relationships.
  • Attribute-based IR offers nuanced relevance: Moving beyond simple keyword matching, evaluating IR relevance based on predefined attributes provides a more granular and informative way to find similar incidents.
  • Low-resource languages benefit from specialized datasets: Creating a Japanese dataset facilitates AI development in this specific language and domain, potentially serving as a foundation for transfer learning to other languages.

Key Results

  • NER: The CNN-Nested-NER model achieved the best performance with a micro F-score of 87.53 and a macro F-score of 84.54.
  • CE: BERT-QA, trained on 700 samples, achieved a strong average F-score of 71.85. When fine-tuned on a refined set of 467 samples with cause-effect pairs, BERT-QA achieved an average F-score of 78.66.
  • IR: The fine-tuned BERT encoder (BERT-finetuned) outperformed the public BERT model and the OpenAI embedding model, achieving the best Recall@k (42.51) and nDCG@k (56.11).

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the dataset size (970 samples for NER/CE, 2,159 for IR) is relatively small, which might require data augmentation or transfer learning for more complex models, especially LLMs.
  • Language Barrier: The dataset is in Japanese, necessitating translation for broader adoption in other languages.
  • Model Performance: While promising, the performance of the models, particularly for CE, can still be improved with stronger architectures and more extensive training.

Future work includes adapting the dataset to create AI pipelines for preventing failures and potentially translating the dataset to more widely used languages.

Summary updated

BibTeX
@inproceedings{inoue2023towards,
  author = {Shumpei Inoue and Minh-Tien Nguyen and Hiroki Mizokuchi and Tuan-Anh D. Nguyen and Huu-Hiep Nguyen and Dung Le},
  title = {{Towards Safer Operations: An Expert-involved Dataset of High-Pressure Gas Incidents for Preventing Future Failures}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-industry.49},
}
Figure