A Framework of Lightweight Deep Cross-Connected Convolution Kernel Mapping Support Vector Machines

Qi Wang, Zhaoying Liu, Ting Zhang, Shanshan Tu, Yujian Li, Muhammad WaqasJournal on Artificial Intelligence 2022 · DOI

Abstract

Deep kernel mapping support vector machines have achieved good results in numerous tasks by mapping features from a low-dimensional space to a high-dimensional space and then using support vector machines for classification. However, the depth kernel mapping support vector machine does not take into account the connection of different dimensional spaces and increases the model parameters. To further improve the recognition capability of deep kernel mapping support vector machines while reducing the number of model parameters, this paper proposes a framework of Lightweight Deep Convolutional Cross-Connected Kernel Mapping Support Vector Machines (LC-CKMSVM). The framework consists of a feature extraction module and a classification module. The feature extraction module first maps the data from low-dimensional to high-dimensional space by fusing the representations of different dimensional spaces through cross-connections; then, it uses depthwise separable convolution to replace part of the original convolution to reduce the number of parameters in the module; The classification module uses a soft margin support vector machine for classification. The results on 6 different visual datasets show that LC-CKMSVM obtains better classification accuracies on most cases than the other five models.

Introduction

Support Vector Machines (SVMs) have long been a cornerstone of classification due to their strong theoretical foundations and generalization abilities. However, their performance traditionally relies on the manual selection of kernel functions to map data into higher-dimensional spaces. While "Deep Kernel Mapping SVMs" have emerged to automate this process using neural networks, they often face two major hurdles: information silos between layers and high computational costs. This paper introduces the Lightweight Deep Convolutional Cross-Connected Kernel Mapping Support Vector Machine (LC-CKMSVM) to bridge these gaps by fusing multi-scale features and employing parameter-efficient convolutions.

Methodology

The LC-CKMSVM framework is divided into a feature extraction module and a classification module. The architecture is designed to capture both high-level semantic features and lower-level structural details through a unique cross-connectivity approach.

The overall architecture of the LC-CKMSVM framework, highlighting the cross-connection and concatenation layer.

Lightweight Feature Extraction

To make the model suitable for deployment on resource-constrained hardware, the authors utilize Depthwise Separable Convolutions (DSConv). Unlike standard convolutions, DSConv splits the operation into depthwise and pointwise stages, drastically reducing the mathematical operations required without significant loss in representational power.

Comparison between (a) standard convolution and (b) depthwise separable convolution (DSConv) used to reduce model parameters.

Cross-Connectivity and Classification

Instead of a simple linear stack of layers, the model employs a concatenation layer that fuses the output of an earlier convolutional block with a later depthwise module. This ensures the final classifier has access to a richer feature set. These fused features are then fed into a Soft Margin SVM. By maximizing the margin between classes, the SVM often achieves better generalization than standard Softmax layers, particularly on smaller or noisier datasets.

Experimental Results

The framework was evaluated across six diverse visual datasets, including gender classification (LFW, FERET, CelebA) and character recognition (EMNIST, KMNIST, SVHN).

  • Accuracy Gains: On the LFW dataset, LC-CKMSVM achieved 91.66% accuracy, outperforming standard CNNs (90.68%) and traditional RBF-Kernel SVMs (84.27%).
  • Parameter Efficiency: The model is remarkably lean. For gender classification, it reduced the parameter count from 124.7K (in the baseline C-CKMSVM) to just 66.0K—a nearly 50% reduction—while simultaneously improving accuracy.
  • Robustness in Character Recognition: On the challenging SVHN dataset, it achieved 88.80% accuracy, significantly outperforming traditional RBFSVMs (32.71%).

Conclusion

LC-CKMSVM demonstrates that "lightweighting" a model does not have to result in a trade-off in accuracy. By combining the efficiency of depthwise separable convolutions with the robust margin-maximization of SVMs and cross-layer feature fusion, the authors have created a powerful hybrid framework. Future work will explore more sophisticated fusion strategies and the application of this framework to more complex tasks like object detection.

Summary updated

Abstract

Deep kernel mapping support vector machines have achieved good results in numerous tasks by mapping features from a low-dimensional space to a high-dimensional space and then using support vector machines for classification. However, the depth kernel mapping support vector machine does not take into account the connection of different dimensional spaces and increases the model parameters. To further improve the recognition capability of deep kernel mapping support vector machines while reducing the number of model parameters, this paper proposes a framework of Lightweight Deep Convolutional Cross-Connected Kernel Mapping Support Vector Machines (LC-CKMSVM). The framework consists of a feature extraction module and a classification module. The feature extraction module first maps the data from low-dimensional to high-dimensional space by fusing the representations of different dimensional spaces through cross-connections; then, it uses depthwise separable convolution to replace part of the original convolution to reduce the number of parameters in the module; The classification module uses a soft margin support vector machine for classification. The results on 6 different visual datasets show that LC-CKMSVM obtains better classification accuracies on most cases than the other five models.
BibTeX
@article{wang2022framework,
  author = {Qi Wang and Zhaoying Liu and Ting Zhang and Shanshan Tu and Yujian Li and Muhammad Waqas},
  title = {{A Framework of Lightweight Deep Cross-Connected Convolution Kernel Mapping Support Vector Machines}},
  journal = {Journal on Artificial Intelligence},
  year = {2022},
  doi = {10.32604/jai.2022.027875},
}
Figure