ChemReco: automated recognition of hand-drawn carbon–hydrogen–oxygen structures using deep learning

Hengjie Ouyang, Wei Liu, Jiajun Tao, Yanghong Luo, Wanjia Zhang, Jiayu Zhou, Shuqi Geng, Chengpeng ZhangScientific 2024 · DOI

TL;DR

This paper introduces ChemReco, a deep learning system for automatically recognizing hand-drawn chemical structures (specifically C, H, and O atoms) and converting them into machine-readable SMILES codes. By developing a novel synthetic data generation method and employing an EfficientNet-Transformer encoder-decoder architecture, they achieve a high accuracy of 96.90%, addressing a critical need for efficient chemical data processing and automated educational tools.

What Problem Does This Paper Address?

In chemistry, hand-drawn molecular structures are a fundamental way to communicate complex information. However, converting these visual representations into a digital, machine-readable format like SMILES is a laborious and error-prone manual process. This bottleneck hinders efficient data analysis, retrieval, and integration into computational workflows. Furthermore, the rise of educational technology has created a demand for automated grading systems that can assess the correctness of student-drawn chemical structures. Existing methods for Optical Chemical Structure Recognition (OCSR) often rely on rule-based systems that struggle with the inherent variability, noise, and complexity of hand-drawn images. There's a significant gap in research and available datasets specifically for recognizing hand-drawn chemical structures, making it difficult to train robust deep learning models.

Methods

The core of this work lies in two main areas: dataset generation and model architecture.

Dataset Generation: * Synthetic Image Generation: To overcome the scarcity of real hand-drawn data, the authors propose a multi-stage synthetic image generation pipeline. This starts with collecting SMILES codes from public databases (PubChem, ZINC, GDB-11, GDB-13) for C, H, and O-containing molecules with up to one ring. These SMILES are then used to generate initial molecular structures using RDKit. * Augmentation and Degradation: To mimic the characteristics of hand-drawn images, these synthetic structures undergo various augmentations (rotation, blur, erosion, dilation, etc.) and degradation processes (simulating loss and deformation). * Background Addition: Random backgrounds are added to further increase variability and robustness. * Diffusion Model Enhancement: A stable diffusion model is used to further refine synthetic images, improving their resemblance to real hand-drawn structures.

Model Architecture: * Encoder-Decoder Framework: The system employs an encoder-decoder architecture, a standard approach for image-to-sequence tasks. * Encoder: EfficientNet is chosen as the image encoder. It's known for its efficiency and strong performance in image classification. The authors leverage transfer learning, using pre-trained weights from ImageNet and fine-tuning them on their chemical structure dataset. * Decoder: A Transformer-based decoder is used. Transformers excel at sequence modeling due to their self-attention mechanism, allowing them to capture long-range dependencies in the generated SMILES string. * Comparison: The authors compare EfficientNet with ResNet as encoders and Transformer with LSTM as decoders, ultimately finding the EfficientNet + Transformer combination to be superior.

Datasets

The study utilizes a combination of real and synthetic datasets:

  • Real Hand-Drawn Data: A dataset of 2,598 images was created by multiple volunteers drawing chemical structures.
  • Synthetic Data: The primary dataset is generated using the proposed synthetic image pipeline. This includes various augmentations, degradations, and background additions. The paper explores the impact of different synthetic generation methods (RDKit, RDKit-aug, RDKit-aug-deg, RDKit-aug-bkg, RDKit-aug-bkg-deg) and varying dataset sizes (100,000 to 1,000,000 images).
  • Real Hand-Drawn Verification Set: For final evaluation, a verification set of real hand-drawn images is used to assess the model's performance on unseen, authentic data.

Key Insights and Ideas

  • Synthetic Data is Crucial: The scarcity of real hand-drawn chemical structure data is a major bottleneck. The proposed synthetic data generation pipeline is essential for creating large, diverse datasets that enable effective deep learning model training.
  • Augmentation and Degradation Matter: Simply generating basic synthetic structures isn't enough. Applying realistic augmentations and degradation processes significantly improves the model's ability to generalize to the variations found in actual hand-drawn images.
  • Backgrounds Can Be Tricky: While adding backgrounds can improve accuracy by providing more context, it can also lead to interference if not handled carefully. The study shows that for background-free datasets, certain degradation methods (RDKit-aug-deg) perform better, while for datasets with backgrounds, methods incorporating backgrounds (RDKit-aug-bkg-deg) yield higher accuracy.
  • Encoder-Decoder with Transformer is Powerful: The combination of EfficientNet as an encoder and Transformer as a decoder proves to be highly effective for this image-to-sequence task, outperforming other combinations.
  • Hybrid Training is Optimal: A mix of synthetic and real hand-drawn images in the training set is superior to using either exclusively. A 90:10 ratio of synthetic to real data achieved the highest accuracy, suggesting that synthetic data provides breadth, while a small amount of real data helps fine-tune for authenticity.

Key Results

  • Overall Accuracy: The proposed ChemReco model, using an EfficientNet + Transformer architecture with a 90:10 synthetic-to-real training data mix, achieved a final recognition accuracy of 96.90% on the test set.
  • Synthetic Data Performance: Different synthetic image generation methods showed varying performance. Methods incorporating augmentation and degradation generally outperformed simpler methods.
  • Dataset Size Impact: Increasing the number of synthetic images in the training set consistently improved accuracy, up to 1 million images.
  • Encoder-Decoder Comparison: EfficientNet + Transformer achieved the highest average exact matching rate (96.90%), outperforming ResNet + LSTM (93.81%), ResNet + Transformer (94.76%), and EfficientNet + LSTM (96.31%).
  • Comparison to Prior Work: The model's accuracy (96.90%) significantly surpasses that of a related study using CNN+LSTM for hydrocarbon structures (76%), highlighting the effectiveness of the proposed approach.

Limitations and Future Work

  • Limited Atom Types: The current model is designed for structures containing only Carbon (C), Hydrogen (H), and Oxygen (O) atoms.
  • Ring Complexity: The dataset construction focused on structures with up to one ring.
  • Future Work: The authors suggest expanding the model to include more heteroatoms and complex ring structures. They also propose further processing of identified SMILES codes to create comprehensive databases and explore applications in educational tools for automated grading and correctness assessment of hand-drawn chemical structures.

Summary updated

Abstract

Chemical molecular structures are a direct and convenient means of expressing chemical knowledge, playing a vital role in academic communication. In chemistry, hand drawing is a common task for students and researchers. If we can convert hand-drawn chemical molecular structures into machine-readable formats, like SMILES encoding, computers can efficiently process and analyze these structures, significantly enhancing the efficiency of chemical research. Furthermore, with the progress of educational technology, automated grading is gaining popularity. When machines automatically recognize chemical molecular structures and assess the correctness of the drawings, it offers great convenience to teachers. We created ChemReco, a tool designed to identify chemical molecular structures involving three atoms: C, H, and O, providing convenience for chemical researchers. Currently, there are limited studies on hand-drawn chemical molecular structures. Therefore, the primary focus of this paper is constructing datasets. We propose a synthetic image method to rapidly generate images resembling hand-drawn chemical molecular structures, enhancing dataset acquisition efficiency. Regarding model selection, the hand-drawn chemical molecule structural recognition model developed in this article achieves a final recognition accuracy of 96.90%. This model employs the encoder-decoder architecture of EfficientNet + Transformer, demonstrating superior performance compared to other encoder-decoder combinations.
BibTeX
@inproceedings{ouyang2024chemreco,
  author = {Hengjie Ouyang and Wei Liu and Jiajun Tao and Yanghong Luo and Wanjia Zhang and Jiayu Zhou and Shuqi Geng and Chengpeng Zhang},
  title = {{ChemReco: automated recognition of hand-drawn carbon–hydrogen–oxygen structures using deep learning}},
  booktitle = {Scientific},
  year = {2024},
  doi = {10.1038/s41598-024-67496-7},
}
Figure