SAMN: A Sample Attention Memory Network Combining SVM and NN in One Architecture

Qiaoling Yang, Linkai Luo, Haoyu Zhang, H. Peng, Ziyang Chen

Abstract

Support vector machine (SVM) and neural networks (NN) have strong complementarity. SVM focuses on the inner operation among samples while NN focuses on the operation among the features within samples. Thus, it is promising and attractive to combine SVM and NN, as it may provide a more powerful function than SVM or NN alone. However, current work on combining them lacks true integration. To address this, we propose a sample attention memory network (SAMN) that effectively combines SVM and NN by incorporating sample attention module, class prototypes, and memory block to NN. SVM can be viewed as a sample attention machine. It allows us to add a sample attention module to NN to implement the main function of SVM. Class prototypes are representatives of all classes, which can be viewed as alternatives to support vectors. The memory block is used for the storage and update of class prototypes. Class prototypes and memory block effectively reduce the computational cost of sample attention and make SAMN suitable for multi-classification tasks. Extensive experiments show that SAMN achieves better classification performance than single SVM or single NN with similar parameter sizes, as well as the previous best model for combining SVM and NN.

Introduction

In the landscape of machine learning, Support Vector Machines (SVM) and Neural Networks (NN) represent two distinct but complementary philosophies. SVMs excel at identifying optimal decision boundaries by analyzing the relationships (inner products) between different samples—essentially looking "outward." Conversely, NNs look "inward," focusing on the complex, non-linear interactions between features within a single sample.

Despite their theoretical synergy, achieving a "true integration" has remained elusive. Previous hybrid attempts typically followed one of two paths: a pipeline approach, where a NN extracts features for a separate SVM classifier, or an optimization approach that reformulates SVMs into a NN-compatible format, often at the cost of the SVM's maximum-margin benefits. Furthermore, traditional SVMs struggle with large-scale data and are inherently binary, making multi-class tasks computationally expensive. SAMN (Sample Attention Memory Network) addresses these challenges by reframing SVM logic as a modular component within a unified neural architecture.

The SAMN Architecture

SAMN integrates the core logic of an SVM as a plug-and-play module. The architecture is designed to capture both inter-sample relationships and intra-feature patterns simultaneously.

The overall architecture of SAMN, illustrating the flow from input through feature extraction, sample attention, and the memory block to the final dual-loss calculation.

Sample Attention Module

A key insight of this work is viewing the SVM as a "sample attention machine." In standard attention mechanisms, models learn to focus on specific parts of an input sequence. In SAMN, the Sample Attention (SA) module allows the network to focus on the most informative samples—analogous to how an SVM identifies support vectors. By treating the SVM classification function as a weighted sum of sample similarities, the researchers can implement SVM logic using the mathematical framework of attention.

Class Prototypes and Memory Block

To overcome the computational bottleneck of comparing every sample in a dataset (a common SVM limitation), SAMN utilizes "class prototypes." These prototypes are mean representations of all samples within a specific category, serving as efficient alternatives to support vectors.

Because neural networks are trained in mini-batches, the model cannot observe every class prototype simultaneously. To bridge this gap, SAMN employs an RNN-style Memory Block (MB). This block stores and updates class prototypes over time, maintaining a "global" understanding of the data distribution even when the model is only processing "local" mini-batches.

The Memory Block (MB) architecture, showing how class prototypes (S) and hidden states (h) are updated over time to maintain global data context.

Dual Loss Function

The training process is guided by a dual loss function designed to optimize the feature space. It minimizes Inner-class distance (ensuring samples are close to their respective class prototypes) while maximizing Inter-class distance (pushing different class prototypes as far apart as possible), mirroring the margin-maximization goal of traditional SVMs.

Experimental Evaluation

Datasets

The researchers evaluated SAMN across 16 tabular datasets from the LIBSVM and UCI repositories, covering a wide range of scales and dimensions:

  • 10 Binary Datasets: Including benchmarks such as Parkinsons, Sonar, Heart, Ionosphere, and Mushrooms.
  • 6 Multi-class Datasets: Including Iris, Wine, Dry Bean, and Accelerometer data.

Performance Comparison

SAMN consistently outperformed standalone SVMs, standard fully connected NNs (CENet), and existing hybrid models like DNMSVM.

In binary classification tasks, SAMN achieved the highest accuracy on 8 out of 10 datasets. Notably, on the Spectf dataset, SAMN reached 81.11% accuracy, a significant improvement over the standard NN's 77.04%. In multi-class scenarios, the advantages were even more pronounced; SAMN outperformed standard NNs across all metrics (Accuracy, Precision, Recall, and F1-score) for all six datasets. For the Wine dataset, SAMN achieved 98.89% accuracy compared to 95.56% for the baseline NN.

Key Insights and Discussion

  • Bridging the Batch Gap: The use of an RNN-based memory block is a critical engineering solution. It allows the model to retain a global memory of class representations, solving the fundamental conflict between the global requirements of SVMs and the local nature of mini-batch NN training.
  • Ablation Studies: To verify the contribution of each component, the researchers tested versions of the model without the attention module (MBN) and without the memory block (SAN). The full SAMN architecture performed best in nearly every instance, confirming that the integration of both sample attention and memory-based prototypes is essential for peak performance.
  • Efficiency: By utilizing prototypes rather than raw support vectors, SAMN remains computationally feasible for large-scale and multi-class tasks where traditional SVMs often become impractical.

Conclusion and Future Work

SAMN represents a successful "true integration" of SVM and NN principles. By reframing sample relationships through the lens of attention and memory, it provides a powerful tool for tabular data classification. While currently focused on tabular data and relatively shallow architectures, future work will explore "Deep SAMN" variants and evaluate the model's performance on unstructured data such as images and audio. Additionally, further research will investigate the scalability of the memory block on even larger, more diverse datasets to ensure its robustness in high-dimensional environments.

Summary updated

Abstract

Support vector machine (SVM) and neural networks (NN) have strong complementarity. SVM focuses on the inner operation among samples while NN focuses on the operation among the features within samples. Thus, it is promising and attractive to combine SVM and NN, as it may provide a more powerful function than SVM or NN alone. However, current work on combining them lacks true integration. To address this, we propose a sample attention memory network (SAMN) that effectively combines SVM and NN by incorporating sample attention module, class prototypes, and memory block to NN. SVM can be viewed as a sample attention machine. It allows us to add a sample attention module to NN to implement the main function of SVM. Class prototypes are representatives of all classes, which can be viewed as alternatives to support vectors. The memory block is used for the storage and update of class prototypes. Class prototypes and memory block effectively reduce the computational cost of sample attention and make SAMN suitable for multi-classification tasks. Extensive experiments show that SAMN achieves better classification performance than single SVM or single NN with similar parameter sizes, as well as the previous best model for combining SVM and NN. The sample attention mechanism is a flexible module that can be easily deepened and incorporated into neural networks that require it.
BibTeX
@misc{yang2023samn,
  author = {Qiaoling Yang and Linkai Luo and Haoyu Zhang and H. Peng and Ziyang Chen},
  title = {{SAMN: A Sample Attention Memory Network Combining SVM and NN in One Architecture}},
  year = {2023},
  eprint = {2309.13930},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.13930},
}
Figure