Continuous-Discrete Convolution for Geometry-Sequence Modeling in Proteins

Hehe Fan, Zhangyang Wang, Yi Yang, Mohan KankanhalliICLR 2023 · OpenReview

TL;DR

Proteins are simultaneously 1D sequences and 3D geometric shapes, but most deep learning models struggle to integrate these two "incompatible" views. This paper introduces Continuous-Discrete Convolution (CDConv), a novel operation that treats the protein chain as a discrete sequence while treating its spatial coordinates as continuous variables. By unifying these perspectives, CDConv achieves state-of-the-art results across protein fold classification and function prediction tasks, outperforming even some massive pretrained models.

What Problem Does This Paper Address?

Proteins are the workhorses of biology, and their function is dictated by their structure. This structure has a dual nature:

  1. 1D Sequence: Amino acids are linked in a regular, linear chain. This structure is discrete and regular.
  2. 3D Geometry: The chain folds into a complex shape where amino acids are distributed unevenly in space. This structure is continuous and irregular.
Protein (3+1)D structure combining regular 1D sequence and irregular 3D geometry.

Existing methods usually fall into two traps. They either model these structures separately (e.g., using an LSTM for the sequence and a GNN for the geometry) and lose the local correlation between them, or they treat both as irregular point clouds, which ignores the helpful "regularity" of the 1D protein backbone. Furthermore, the units of length in sequence space (indices) and geometric space (Angstroms) are incompatible, making simple fusion difficult.

Methods

The authors propose Continuous-Discrete Convolution (CDConv) to bridge this gap.

Conceptual illustration of Continuous-Discrete Convolution.
  • The Hybrid Kernel: Unlike a standard 1D convolution (which uses a fixed weight for each neighbor in a sequence) or a continuous convolution (which uses a neural network to generate weights based on 3D distance), CDConv does both. It assigns independent learnable weights to different discrete sequential displacements (e.g., the neighbor 2 positions away in the chain) but modulates those weights by directly encoding the continuous geometric displacements (the 3D distance and orientation).
  • Rotation Invariance: To ensure the model doesn't get confused if a protein is rotated in space, they use a "relative spatial encoding." This describes the position of a neighbor using a local coordinate system based on the protein's backbone, rather than absolute $(x, y, z)$ coordinates.
  • (3+1)D Hierarchical Architecture: They build a ResNet-like "pyramid" network. As the data moves deeper into the network, the protein chain is downsampled (pooling), and the 3D "receptive field" (the radius the model looks at) is progressively enlarged. This allows the model to capture local motifs early on and global topology later.

Datasets

The model was evaluated on four major protein benchmarks:

  1. Protein Fold Classification: Using the SCOPe 1.75 dataset, tested at the Fold, Superfamily, and Family levels.
  2. Enzyme Reaction Classification: Predicting the catalytic reaction of enzymes across 384 classes.
  3. Gene Ontology (GO) Term Prediction: Multi-label classification for Biological Process (BP), Molecular Function (MF), and Cellular Component (CC).
  4. Enzyme Commission (EC) Number Prediction: Predicting specific functional numbers for enzymes.

Key Insights and Ideas

  • Regularity Matters: The paper demonstrates that the 1D order of a protein is a powerful signal. By using discrete weights for sequential neighbors, CDConv captures the "grammar" of the protein chain more effectively than models that treat the sequence as just another spatial attribute.
  • The "Core" Hypothesis: Through visualization of the network's activations, the authors found that early layers focus heavily on the central regions (the core) of the protein.
  • Aha Moment: To test this, they performed "masking" experiments. Removing 20% of the amino acids from the center of the protein caused a much larger drop in accuracy than removing 20% from the surface. This suggests that the internal packing of a protein contains more discriminative information for its identity and function than its surface area.
  • Efficiency vs. Scale: Remarkably, CDConv (trained from scratch) outperformed several massive models like ESM-1b and ProtBERT—which are pretrained on millions of sequences—on specific fold and reaction classification tasks.

Key Results

  • Fold Classification: On the "Superfamily" level, CDConv achieved 77.7% accuracy, a significant jump over the previous state-of-the-art (GearNet-Edge-IEConv at 70.3%).
  • Function Prediction: In GO-BP term prediction, it reached an $F_{max}$ of 0.453, outperforming the best previous residue-level model (0.421).
  • Ablation Success: The authors showed that a "two-stream" approach (modeling 1D and 3D separately and then fusing) is significantly inferior to their unified (3+1)D approach, proving that the local interplay between sequence and geometry is vital.

Limitations and Future Work

  • Rigid Hyperparameters: CDConv requires a predefined 3D radius and 1D kernel size. These are fixed for the whole model, which might be "rigid" for proteins of vastly different sizes or densities.
  • Attention Integration: The authors suggest that replacing fixed-radius neighborhoods with self-attention could allow the model to adaptively focus on the most important neighbors, regardless of distance.
  • Biological Validation: The finding that the "central region" is more important is a computational observation. The authors note that this needs to be verified through actual biochemical experiments in the future.
  • Missing Data: While the model handles missing coordinates better than others, real-world protein data (like that from Cryo-EM) often has significant noise or missing loops that still pose a challenge.

Summary updated

Abstract

The structure of proteins involves 3D geometry of amino acid coordinates and 1D sequence of peptide chains. The 3D structure exhibits irregularity because amino acids are distributed unevenly in Euclidean space and their coordinates are continuous variables. In contrast, the 1D structure is regular because amino acids are arranged uniformly in the chains and their sequential positions (orders) are discrete variables. Moreover, geometric coordinates and sequential orders are in two types of spaces and their units of length are incompatible. These inconsistencies make it challenging to capture the 3D and 1D structures while avoiding the impact of sequence and geometry modeling on each other. This paper proposes a Continuous-Discrete Convolution (CDConv) that uses irregular and regular approaches to model the geometry and sequence structures, respectively. Specifically, CDConv employs independent learnable weights for different regular sequential displacements but directly encodes geometric displacements due to their irregularity. In this way, CDConv significantly improves protein modeling by reducing the impact of geometric irregularity on sequence modeling. Extensive experiments on a range of tasks, including protein fold classification, enzyme reaction classification, gene ontology term prediction and enzyme commission number prediction, demonstrate the effectiveness of the proposed CDConv.
BibTeX
@inproceedings{fan2023continuous,
  author = {Hehe Fan and Zhangyang Wang and Yi Yang and Mohan Kankanhalli},
  title = {{Continuous-Discrete Convolution for Geometry-Sequence Modeling in Proteins}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=P5Z-Zl9XJ7},
}
Figure