TL;DR
ZeoBench is a new benchmark for self-supervised learning (SSL) on 3D volumetric representations of zeolites, porous crystalline materials used in industrial catalysis and gas storage. By pretraining on over 331,000 unlabeled structures, the researchers demonstrate that SSL significantly improves property prediction in low-data regimes and even generalizes to out-of-domain materials like Metal-Organic Frameworks (MOFs).
What Problem Does This Paper Address?
Zeolites are essential "molecular sieves" in the chemical industry, but finding the perfect zeolite for a specific application is a needle-in-a-haystack problem. While we have databases of hundreds of thousands of theoretical zeolite structures, determining their properties (like how well they adsorb a specific hydrocarbon) requires computationally expensive Monte Carlo simulations that can take hours per structure.
This creates a "low-data regime" where we have a massive corpus of unlabeled 3D structures but very few expensive labels. Previous work used supervised 3D Convolutional Neural Networks (CNNs) to predict properties, but these models require thousands of labels to be effective. ZeoBench addresses this gap by investigating how self-supervised learning can leverage the unlabeled 300k+ structures to learn generalizable geometric features, reducing the need for labeled data.
Methods
The authors represent zeolites as 3D Signed Distance Grids, where each voxel encodes the distance to the nearest framework surface. They evaluate four popular SSL objectives:
- Contrastive: SimCLR and MoCo v2+ (learning by distinguishing between different views of the same structure).
- Distillation: BYOL and DINO (learning by matching a student network's output to a teacher network).
A key technical contribution is the exploration of 3D data augmentations. While standard 2D image augmentations (like color jittering) don't apply to distance grids, the authors found that scale augmentation—independently varying the grid resolution along each axis—is critical for learning robust features.
They also provide a rigorous comparison against "ImageNet-to-3D" adaptations, where 2D models pretrained on natural images are adapted to 3D volumes via three strategies:
- 2D Slices: Stacking random 2D slices as RGB channels.
- Channel Inflation: Duplicating 2D weights to match 3D volume depth.
- Depth Tokenization: Treating the 3D volume as a sequence of image patches for a Vision Transformer (ViT).
Datasets
- Pretraining Corpus: 331,172 theoretical and 402 experimentally synthesized all-silica zeolite structures.
- Downstream Tasks: Six molecular adsorption tasks (predicting Henry’s constant for various hydrocarbons like C18, C24, and branched isomers) across varying label budgets (100 to 10,000 samples).
- Transfer Learning: The hMOF dataset, consisting of 137,000 Metal-Organic Frameworks, used to test if zeolite-pretrained features generalize to different chemical domains.
Key Insights and Ideas
- CNNs > Transformers (for now): In this 3D geometric setting, 3D ResNets consistently outperformed Vision Transformers (ViT) and Swin Transformers. The authors suggest that the translational equivariance priors in CNNs are better suited for grid-based molecular data.
- Contrastive SSL Wins on Frozen Features: For "linear probing" (where the backbone is frozen), contrastive methods like SimCLR and MoCo produced the strongest representations. This is likely because their training objective encourages a more uniform distribution of embeddings, which is beneficial for continuous regression tasks like adsorption prediction.
- Scale is the Secret Sauce: Without scale augmentation during pretraining, SSL performance drops significantly. Scaling forces the network to learn global topological features rather than just memorizing local surface curvatures.
- ImageNet is a Surprisingly Strong Baseline: When adapted via "depth tokenization," ImageNet-pretrained features outperformed hand-crafted descriptors and even some medical-domain 3D models, though they still lagged behind in-domain zeolite SSL.
Key Results
The study establishes that in-domain SSL provides a massive boost in performance, particularly when labels are scarce.
- Low-Label Performance: With only 100 labels, SSL-pretrained models achieved significantly lower Mean Squared Error (MSE) compared to training from scratch or using hand-crafted descriptors.
- State-of-the-Art: On the full C18 adsorption task, SSL pretraining improved performance by 16% over the previous best supervised baseline (ZeoNet).
- Cross-Domain Generalization: A SimCLR model pretrained on zeolites outperformed existing MOF-specific architectures (like MOFormer) on the hMOF dataset, proving that the model learns fundamental principles of nanoporous geometry.
Limitations and Future Work
While ZeoBench provides a comprehensive foundation, the authors note several areas for growth:
- Architecture Diversity: The study focused primarily on 3D ResNet-18. Future work could explore larger backbones or group-equivariant networks that natively handle 3D rotations.
- Chemical Complexity: The current benchmark focuses on all-silica zeolites. Expanding to zeolites with different framework atoms (like Aluminum) or extra-framework ions would increase the benchmark's industrial relevance.
- Task Expansion: Beyond adsorption, future versions could include tasks like catalytic activity or synthetic feasibility.
The code and dataset splits are available at https://github.com/cvl-umass/ZeoBench.
Summary updated