EAPCR: A Universal Feature Extractor for Scientific Data without Explicit Feature Relation Patterns

Yu Wu, Zhuohang Yu, Ling An, Yansong Li, Yu Wu 0020, Zeyu Dong, Zhangdi Liu, Le Gao, Zhenyu Zhang, Chichun Zhou

Abstract

Deep learning has revolutionized fields with structured data like computer vision and natural language processing. However, it often underperforms on "tabular" scientific data where explicit Feature Relation Patterns (FRPs) are absent. We introduce EAPCR, a novel architecture that exposes all possible pairwise feature interactions and employs a Permuted CNN to efficiently capture global relationships. EAPCR consistently outperforms traditional machine learning and state-of-the-art deep learning models across diverse scientific domains, bridging the gap between deep learning and heterogeneous scientific datasets.

Introduction: The Challenge of Scientific Data

While deep learning excels in domains with inherent spatial or sequential structures, scientific data—such as medical records or sensor readings—often lacks these explicit Feature Relation Patterns (FRPs). In an image, pixels have a spatial relationship; in text, words follow a sequence. In contrast, features like "Age" and "Blood Pressure" in a medical dataset have no predefined order or distance.

Standard architectures like CNNs or Transformers are biased toward specific FRPs. Without these priors, deep models struggle to identify meaningful interactions, frequently falling behind traditional "shallow" models like XGBoost or Random Forest.

Scientific data lacks the explicit relation patterns found in images or text, making feature extraction difficult for standard deep learning models.

The EAPCR Architecture

To address the lack of explicit FRPs, we propose EAPCR, a three-stage feature extractor designed to uncover hidden relationships:

  1. Embedding and Bilinear Attention: Features are first converted into dense embedding vectors. A correlation matrix is then computed using bilinear attention ($A = \text{Tanh}(EE^T)$). This process "exposes" every possible pairwise interaction between features in a grid format, regardless of their original order.
  2. Permuted CNN: Standard CNNs are limited by local receptive fields. EAPCR introduces a designed permutation matrix to reshuffle the correlation matrix. This brings "far apart" features into close proximity, allowing a lightweight CNN to capture global dependencies efficiently.
  3. Residual MLP Path: To maintain global context, the original embeddings are pooled and passed through a Multi-Layer Perceptron (MLP). This residual connection ensures that the model integrates both specific feature interactions and broader data trends.
The EAPCR workflow: (a) Architecture overview, (b) Exposing FRPs via attention, and (c) Efficient sampling through permutation and CNN.

Experimental Evaluation

EAPCR was evaluated on a diverse set of real-world and synthetic benchmarks, including UCI Heart Disease, $TiO_2$ photocatalysis efficiency, and machine failure prediction. To test robustness against the loss of FRPs, standard datasets (MNIST, CIFAR-10) were "scrambled" by shuffling pixels, creating a controlled environment for data without explicit structure.

Key Results

EAPCR consistently outperformed both traditional machine learning and state-of-the-art deep learning baselines:

  • Medical Diagnostics: On the Heart Disease dataset, EAPCR achieved 93% accuracy, outperforming SVM (85%) and Logistic Regression (83%).
  • Chemical Property Prediction: In catalysis efficiency tasks, EAPCR reached an $R^2$ of 0.937, surpassing LightGBM (0.928) and XGBoost (0.884).
  • Robustness: On scrambled MNIST, EAPCR maintained 94.5% accuracy, while standard CNNs suffered significant performance drops.

Discussion and Insights

The success of EAPCR highlights that FRPs act as essential "priors" for deep learning. By explicitly exposing these patterns and using permutations to expand the receptive field of a CNN, EAPCR effectively "reconstructs" the underlying logic of unstructured datasets. This approach allows for high-performance feature extraction with fewer parameters than large-scale Transformers.

Conclusion and Future Work

EAPCR provides a universal framework for scientific data analysis. Future research will focus on improving the handling of continuous numerical data beyond discretization and optimizing the correlation matrix construction for high-dimensional datasets like genomics.

Summary updated

Abstract

Conventional methods, including Decision Tree (DT)-based methods, have been highly effective in scientific tasks, such as non-image medical diagnostics, system anomaly detection, and inorganic catalysis efficiency prediction. However, most deep-learning techniques have struggled to surpass or even match this level of success as traditional machine learning methods. The primary reason is that these applications involve multi-source, heterogeneous data, where features lack explicit relationships. This contrasts with image data, where pixels exhibit spatial relationships; textual data, where words have sequential dependencies; and graph data, where nodes are connected through established associations. The absence of explicit **F**eature **R**elation **P**atterns (FRPs) presents a significant challenge for deep learning techniques in scientific applications that are not image, text, and graph-based. In this paper, we introduce *EAPCR*, a universal feature extractor designed for data without explicit FRPs. Tested across various scientific tasks, EAPCR consistently outperforms traditional methods and bridges the gap where deep learning models fall short. To further demonstrate its robustness, we synthesize a dataset without explicit FRPs. While Kolmogorov–Arnold Network (KAN) and feature extractors like Convolutional Neural Networks (CNNs), Graph Convolutional Networks (GCNs), and Transformers struggle, EAPCR excels, demonstrating its robustness and superior performance in scientific tasks without FRPs.
BibTeX
@misc{wu2024eapcr,
  author = {Yu Wu and Zhuohang Yu and Ling An and Yansong Li and Yu Wu and Zeyu Dong and Zhangdi Liu and Le Gao and Zhenyu Zhang and Chichun Zhou},
  title = {{EAPCR: A Universal Feature Extractor for Scientific Data without Explicit Feature Relation Patterns}},
  year = {2024},
  eprint = {2411.08164},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.08164},
}
Figure