A ridgelet kernel regression model using genetic algorithm

Shuyuan Yang, Min Wang, Licheng Jiao

Abstract

Approximating complex, high-dimensional functions with spatial inhomogeneities remains a significant challenge in machine learning. Traditional methods like Fourier transforms and wavelets often struggle with singularities in higher dimensions. This paper introduces a Ridgelet Kernel Regression Model that leverages the directional strengths of ridgelets within a kernel machine framework. By employing a Genetic Algorithm (GA) to optimize ridgelet orientations, the model achieves superior performance in representing functions with ridge-like singularities. Evaluations on synthetic singular functions and radar target recognition data show that the proposed method outperforms Gaussian and wavelet-based Support Vector Machines (SVMs) in both accuracy and training efficiency.

Introduction

Multivariate function approximation (MVFA) is essential for applications ranging from industrial control to fault detection. However, two primary obstacles persist: the "curse of dimensionality," where data sparsity in high-dimensional spaces complicates smooth estimation, and "spatial inhomogeneities," where real-world data contains non-smooth patches or singularities. While neural networks and wavelets are common tools, they often suffer from overfitting, slow convergence, or poor scaling in high dimensions. This research proposes a hybrid approach that combines the mathematical robustness of Kernel Machines with the flexibility of ridgelet theory to specifically target these issues.

Methodology

The proposed model integrates three core components to enhance function approximation:

  1. Ridgelet Basis Functions: Unlike standard wavelets, ridgelets are designed to represent functions with "ridge-like" structures. They are particularly effective at capturing singularities along hyperplanes in high-dimensional space.
  2. Kernel Regularization: By embedding ridgelets into a Kernel Machine framework, the authors utilize a regularized Mean Squared Error (MSE) objective function. This provides a stricter mathematical foundation than traditional neural networks, improving generalization and preventing overfitting.
  3. Genetic Algorithm (GA) Optimization: Determining the optimal direction (vector $u$) for ridgelets is a non-trivial optimization problem. The authors use a GA to evolve a population of directional angles, using the reciprocal of the regularized objective function as a fitness score to find the global optimum.
Architecture of the proposed Ridgelet Kernel Regression Model.

Experimental Results

The model's performance was benchmarked against Gaussian kernel LS-SVM (GSVM) and Wavelet kernel LS-SVM (WSVM) using two distinct datasets: synthetic singular functions ($f^1$ through $f^5$) and a radar target recognition dataset featuring 1,084 images of three aircraft types (B52, J-6, and J-7).

Function Approximation

In tests involving synthetic functions with sharp transitions, the ridgelet-based model consistently achieved the lowest Root Mean Squared Error (RMSE). As shown in the performance table, the training errors for the proposed method were several orders of magnitude lower than those of GSVM and WSVM, particularly for functions with complex singularities.

Comparison of RMSE for function approximation across different models on synthetic singular functions.

Radar Target Recognition

For the radar dataset, the ridgelet model maintained high recognition rates (91%–97% on test sets), matching or exceeding the performance of WSVM. A significant advantage was observed in training efficiency; the proposed method required only 1.1 seconds for training, compared to 4.5 seconds for GSVM and 2.8 seconds for WSVM.

Discussion and Conclusion

The results demonstrate that for high-dimensional data with "ridges" or sharp transitions, ridgelet-based kernels are significantly more efficient than standard Gaussian or Wavelet kernels. By using a Genetic Algorithm instead of a greedy search, the model effectively identifies global optimal directions for its basis functions, bridging the gap between harmonic analysis and statistical learning theory.

Future work should address the computational overhead of GAs as dimensionality increases and perform a more detailed sensitivity analysis of hyperparameters like the regularization constant $\lambda$. Additionally, testing the model on larger "Big Data" scales will be crucial for validating its practical industrial applicability.

Summary updated

BibTeX
@inproceedings{yang2005ridgelet,
  author = {Shuyuan Yang and Min Wang and Licheng Jiao},
  title = {{A ridgelet kernel regression model using genetic algorithm}},
  booktitle = {ESANN},
  year = {2005},
}
Figure