Chemical structure recognition method based on attention mechanism and encoder-decoder architecture

Shuiling Zeng, Zhaoxian Li, Jiaxiong Zhang, Longfei Ding, Cairong ZhaoJournal of Image and Graphics 2024 · DOI

Abstract

Chemical structure recognition is a critical task at the intersection of chemistry and computer vision. Traditional Optical Chemical Structure Recognition (OCSR) techniques often suffer from information loss or misidentification in complex tasks due to structural diversity. Deep learning models, while promising, frequently encounter issues with high architectural complexity, loss of contextual information, and low recognition rates. This paper proposes a novel chemical structure recognition method combining an attention mechanism with an encoder-decoder architecture. The approach utilizes a modified ResNet50 for feature extraction, a Bi-directional Long Short-Term Memory (BLSTM) network for row-wise encoding to enhance spatial information, and a coverage-attention-based LSTM decoder to translate images into Simplified Molecular Input Line Entry System (SMILES) sequences. Experimental results across multiple datasets, including CASIA-CSDB and USPTO, demonstrate that the proposed method achieves superior accuracy compared to state-of-the-art models like Image2Mol and ChemPix, even with limited training data.

Introduction

Converting chemical structures from images into machine-readable formats like SMILES strings is essential for digitizing chemical knowledge and accelerating drug discovery. However, traditional rule-based methods struggle with the complexity of molecular layouts and noisy backgrounds. Deep learning approaches offer a solution but often require immense data and struggle with maintaining fine-grained spatial relationships or avoiding repetitive outputs in the decoding phase.

Proposed Methodology

The proposed framework employs a specialized encoder-decoder architecture designed to capture intricate molecular details and maintain global context.

Overall architecture of the proposed chemical structure recognition model showing the flow from input image to SMILES output.

Encoder: Feature Extraction and Spatial Reinforcement

The encoder consists of two primary components:

  1. Modified ResNet50: To preserve high-resolution spatial features necessary for identifying small chemical details (e.g., double bonds, atom labels), the standard ResNet50 is truncated by removing the final two stages and global pooling layers.
  2. Row-wise BLSTM: A Bi-directional LSTM processes the feature maps row-by-row. This step reinforces the spatial connectivity and context of the molecular structure before it is passed to the decoder.
Illustration of the row encoding process where feature maps are processed to strengthen spatial context.

Decoder: Coverage-Attention Mechanism

The decoder uses an LSTM network integrated with a "coverage" attention mechanism. This mechanism tracks previously attended regions of the image, preventing the model from generating redundant characters or skipping parts of the molecule—a common failure mode in standard attention decoders.

Experimental Evaluation

The model was evaluated on a wide range of datasets, including large-scale benchmarks like CASIA-CSDB (~480,000 images) and various external patent and journal datasets (USPTO, ACS, Indigo).

Key Results

The method demonstrated significant performance gains:

  • Accuracy: Achieved an Exact Match (EM) accuracy of 84.42% on CASIA-CSDB and 85.78% on a smaller subset (Mini CASIA-CSDB), indicating high data efficiency.
  • Generalization: On the USPTO patent dataset, it reached an EM of 58.21%, outperforming baseline models like SwimOCSR and ChemPix.
  • Efficiency: The architecture achieves competitive results with significantly fewer training samples than many existing deep learning models.

Conclusion

By combining a high-resolution CNN backbone with row-wise spatial encoding and a coverage-aware decoder, this method provides a robust solution for chemical structure recognition. Future work will focus on improving the recognition of stereochemistry and enhancing robustness against low-quality or highly complex molecular scans.

Summary updated

Abstract

目的化学结构识别是化学和计算机视觉领域的一个重要问题,传统光学化学结构识别技术在复杂化学结构识别任务中易发生信息丢失或误识别的现象,同时又因为化学物质的结构多样性常导致其无法解析,识别效果不佳。而基于深度学习的模型通常具有网络结构复杂度高、上下文信息易丢失和识别率低的问题。为此,提出一种结合注意力机制和编码器—解码器架构的化学结构识别方法。方法首先,使用改进的ResNet50(residual network)作为特征提取器抓取表征信息;其次,使用BLSTM(bi-directional long-short term memory)作为行编码器为ResNet50提取的表征信息加强空间信息;最后,使用去填充模块和基于覆盖注意力机制的LSTM(long short-term memory)网络作为模型解码器,对化学结构图像进行解码,将编码结果解码为SMILES(simplified molecular input line entry system)序列。结果在Indigo、ChemDraw、CLEF(Conference and Labs of the Evaluation Forum)、JPO(Japanese Patent Office)、UOB(University of Birmingham)、USPTO(United States Patent and Trademark Office)、Staker、ACS(American Chemistry Society)、CASIA-CSDB(Institute of Automation of Chinese Academy of Sciences—Chemical Structure Database)和Mini CASIA-CSDB数据集上,所提方法识别准确率分别为71.1%、70.21%、45.8%、30.3%、53.02%、58.21%、43.39%、46.3%、84.42%和85.78%,高于SwimOCSR、Image2Mol和ChemPix模型得分。结论与其他模型相比,本文方法通过少量训练集能够获得较高的识别准确率。
BibTeX
@article{zeng2024chemical,
  author = {Shuiling Zeng and Zhaoxian Li and Jiaxiong Zhang and Longfei Ding and Cairong Zhao},
  title = {{Chemical structure recognition method based on attention mechanism and encoder-decoder architecture}},
  journal = {Journal of Image and Graphics},
  year = {2024},
  doi = {10.11834/jig.230367},
}
Figure