Abstract
Deep learning has revolutionized fields with structured data like computer vision and natural language processing. However, it often underperforms on "tabular" scientific data where explicit Feature Relation Patterns (FRPs) are absent. We introduce EAPCR, a novel architecture that exposes all possible pairwise feature interactions and employs a Permuted CNN to efficiently capture global relationships. EAPCR consistently outperforms traditional machine learning and state-of-the-art deep learning models across diverse scientific domains, bridging the gap between deep learning and heterogeneous scientific datasets.
Introduction: The Challenge of Scientific Data
While deep learning excels in domains with inherent spatial or sequential structures, scientific data—such as medical records or sensor readings—often lacks these explicit Feature Relation Patterns (FRPs). In an image, pixels have a spatial relationship; in text, words follow a sequence. In contrast, features like "Age" and "Blood Pressure" in a medical dataset have no predefined order or distance.
Standard architectures like CNNs or Transformers are biased toward specific FRPs. Without these priors, deep models struggle to identify meaningful interactions, frequently falling behind traditional "shallow" models like XGBoost or Random Forest.
The EAPCR Architecture
To address the lack of explicit FRPs, we propose EAPCR, a three-stage feature extractor designed to uncover hidden relationships:
- Embedding and Bilinear Attention: Features are first converted into dense embedding vectors. A correlation matrix is then computed using bilinear attention ($A = \text{Tanh}(EE^T)$). This process "exposes" every possible pairwise interaction between features in a grid format, regardless of their original order.
- Permuted CNN: Standard CNNs are limited by local receptive fields. EAPCR introduces a designed permutation matrix to reshuffle the correlation matrix. This brings "far apart" features into close proximity, allowing a lightweight CNN to capture global dependencies efficiently.
- Residual MLP Path: To maintain global context, the original embeddings are pooled and passed through a Multi-Layer Perceptron (MLP). This residual connection ensures that the model integrates both specific feature interactions and broader data trends.
Experimental Evaluation
EAPCR was evaluated on a diverse set of real-world and synthetic benchmarks, including UCI Heart Disease, $TiO_2$ photocatalysis efficiency, and machine failure prediction. To test robustness against the loss of FRPs, standard datasets (MNIST, CIFAR-10) were "scrambled" by shuffling pixels, creating a controlled environment for data without explicit structure.
Key Results
EAPCR consistently outperformed both traditional machine learning and state-of-the-art deep learning baselines:
- Medical Diagnostics: On the Heart Disease dataset, EAPCR achieved 93% accuracy, outperforming SVM (85%) and Logistic Regression (83%).
- Chemical Property Prediction: In catalysis efficiency tasks, EAPCR reached an $R^2$ of 0.937, surpassing LightGBM (0.928) and XGBoost (0.884).
- Robustness: On scrambled MNIST, EAPCR maintained 94.5% accuracy, while standard CNNs suffered significant performance drops.
Discussion and Insights
The success of EAPCR highlights that FRPs act as essential "priors" for deep learning. By explicitly exposing these patterns and using permutations to expand the receptive field of a CNN, EAPCR effectively "reconstructs" the underlying logic of unstructured datasets. This approach allows for high-performance feature extraction with fewer parameters than large-scale Transformers.
Conclusion and Future Work
EAPCR provides a universal framework for scientific data analysis. Future research will focus on improving the handling of continuous numerical data beyond discretization and optimizing the correlation matrix construction for high-dimensional datasets like genomics.
Summary updated