ZSEE: A Dataset based on Zeolite Synthesis Event Extraction for Automated Synthesis Platform

Song He, Xin Peng, Yihan Cai, Xin Li, Zhiqing Yuan, Wenli Du, Weimin YangNAACL 2024 · DOI

Introduction

The chemical industry is increasingly moving toward "autonomous unmannedness," where robotic platforms execute experiments based on structured synthesis plans. However, the vast majority of chemical knowledge remains trapped in natural language text within millions of research papers and patents. Zeolite synthesis, a critical process for industrial catalysis, presents a significant challenge for automated extraction due to its procedural complexity. To bridge this "semantic gap," researchers have introduced ZSEE, the first large-scale dataset specifically designed for extracting fine-grained event data from zeolite synthesis literature.

Overview of the Event Extraction task in zeolite synthesis literature, showing the mapping from unstructured text to structured event triggers and arguments.

The ZSEE Dataset

The ZSEE (Zeolite Synthesis Event Extraction) dataset consists of nearly 5,000 sentences extracted from over 1,000 scientific documents. It contains 8,873 annotated events and 14,331 synthesis arguments. The schema is highly detailed, featuring 16 event types (e.g., Crystallize, Wash, Calcine, Set PH) and 13 argument roles (e.g., solvent, revolution, pressure).

The dataset's quality is ensured by a rigorous annotation process involving 11 chemistry and computer science graduate students. The inter-annotator agreement, measured by a Fleiss’ Kappa of 0.89, indicates high reliability. Compared to general-purpose datasets like ACE2005 or other domain-specific ones like PHEE, ZSEE is significantly more "dense," capturing more events and arguments per sentence.

Methodology

The authors frame the extraction as an Event Extraction (EE) task consisting of two sub-tasks: Event Detection (ED), which identifies the action "trigger" (e.g., mixed), and Event Argument Extraction (EAE), which identifies the associated details (e.g., material, temperature).

Two primary architectural approaches were benchmarked:

  1. Classification-based (Extractive QA): Utilizing BERT models to treat extraction as a question-answering task.
  2. Generation-based: Utilizing BART models to generate structured sentences based on predefined templates.
Comparison of classification-based (BERT) and generation-based (BART) extraction methods for identifying triggers and arguments.

Advanced EAE methods such as PAIE and AMPERE were also evaluated to better capture the relationships between actions and their properties using role-specific selectors and abstract meaning representations.

Results and Discussion

The experimental results highlight several key findings:

  • Triggers are "Easy," Arguments are "Hard": Models excel at identifying actions (92%+ F1 score) due to standardized chemical verbs. However, identifying specific details is more challenging (~74% F1) because chemical names and conditions are linguistically complex and often nested.
  • The "Long Sentence" Penalty: Performance degrades as sentence length increases. Zeolite literature often contains multiple steps and conditions in a single sentence, creating long-distance dependencies that are difficult for models to map.
  • Template Power: Generation-based models (like PAIE using BART-large) outperformed classification-based models in argument extraction (74.17% F1). The templates provide a structural hint that helps the model organize the extracted data.
  • Data Imbalance: While models perform exceptionally well on frequent events like Stir (96% Tri-C), they struggle with rare events like Sonicate (80% Tri-C), indicating a low-resource challenge within the domain.

Conclusion

ZSEE provides a critical foundation for transforming unstructured experimental narratives into machine-readable "recipes" for automated synthesis. While current models show promise, hurdles remain regarding abstract expressions, event ordering, and complex chemical entities. Future research will focus on data augmentation for rare events and document-level extraction to capture full synthesis workflows across multiple sentences.

Summary updated

BibTeX
@inproceedings{he2024zsee,
  author = {Song He and Xin Peng and Yihan Cai and Xin Li and Zhiqing Yuan and Wenli Du and Weimin Yang},
  title = {{ZSEE: A Dataset based on Zeolite Synthesis Event Extraction for Automated Synthesis Platform}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.116},
}
Figure