Manually Selecting The Data Function for Supervised Learning of small datasets

Amir Khanjari, Saeid Pourmand, Mohammad Reza FaridrohaniarXiv 2023 · OpenReview

Abstract

Supervised learning problems often become ill-posed when data is scarce, leading to unstable and non-unique solutions. Rather than relying solely on traditional regularization, this research proposes initializing an informative ill-posed operator to "pose better questions." By utilizing the Fredholm Integral Equation of the First Kind (FIFK), we can integrate input distributions and prior knowledge. We introduce the Manually Selecting Data Function (MSDF) method, which allows for different kernels on both sides of the equation. Experiments on real-world datasets demonstrate that this approach achieves higher accuracy and stability compared to standard least-squares and other semi-supervised methods.

Introduction: The Challenge of Small Data

In specialized fields such as material science, neuroimaging, and biomedical engineering, researchers frequently encounter datasets with limited samples. In these scenarios, supervised learning becomes an "ill-posed" problem. The underlying data-generating process (DGP) cannot be accurately estimated, making models prone to overfitting and highly sensitive to noise.

Traditional solutions typically involve Tikhonov regularization, which penalizes model complexity to stabilize the solution. However, this paper argues for a shift in perspective: instead of just penalizing the model, we should refine the mathematical framework used to define the relationship between features.

Methodology: Leveraging Fredholm Integral Equations

The core of this research lies in the Fredholm Integral Equation of the First Kind (FIFK). This mathematical structure is a reliable ill-posed operator capable of incorporating prior knowledge and input distributions. The authors explore two primary implementations:

1. Semi-Supervised V-Matrix (SV-M)

Building upon Vapnik’s V-Matrix method, SV-M utilizes unlabeled data to better estimate the input distribution. By including these points, the model gains a clearer understanding of the "distributional location" of training samples, enhancing robustness against noise.

2. Manually Selecting Data Function (MSDF)

MSDF is the primary contribution of this work. Standard Fredholm setups often use the same kernel for both the mapping operator and the data function. MSDF breaks this symmetry, allowing researchers to manually select different kernels (e.g., Gaussian, Laplacian, or Bessel) for each side. This asymmetry provides a more effective means of controlling the high-dimensional integral equation, preventing the model from overfitting to the specific noise patterns of small datasets.

Experimental Evaluation

The methods were benchmarked against 12 binary classification datasets from the UCI Machine Learning Repository, including Abalone, Breast Cancer, and Sonar. These datasets ranged from 208 to 4177 samples with 4 to 61 attributes, providing a rigorous test for handling limited training proportions.

Performance and Stability

The MSDF method consistently outperformed established techniques such as Kernel Regularized Least Square (KRLS) and Manifold Regularization (MR). Notably, on the "Sonar" dataset, MSDF maintained high accuracy even when trained on only 1% of the available data.

A critical advantage of the proposed methods is their stability. Unlike standard regularized least-squares, where performance can fluctuate wildly based on the choice of the regularization parameter ($\lambda$), the V-Matrix and MSDF approaches maintain consistent accuracy and parameter values.

Comparison of parameter value and RMSE stability between Regularized Least-Squares and the V-Matrix method across varying Lambda values.

As shown in the figure above, the V-Matrix method (representative of the operator-based approach) exhibits a flat response to changes in $\lambda$, whereas the standard least-squares method shows significant sensitivity, with RMSE increasing and parameter values decaying as regularization increases.

Conclusion and Future Work

While the MSDF method offers significant gains in accuracy and stability for small datasets, it comes with computational trade-offs. Calculating SV-Matrices is intensive ($O(m^2nd^2)$), and selecting the optimal combination of kernels requires robust cross-validation. Future research will focus on applying these distribution-dependent operators to domain adaptation and improving robustness against covariate shifts.

By treating the structure of the integral operator itself as a form of regularization, this work provides a powerful new tool for machine learning in information-poor environments.

Summary updated

Abstract

Supervised learning problems may become ill-posed when there is a lack of information, resulting in unstable and non-unique solutions. However, instead of solely relying on regularization, initializing an informative ill-posed operator is akin to posing better questions to achieve more accurate answers. The Fredholm integral equation of the first kind (FIFK) is a reliable ill-posed operator that can integrate distributions and prior knowledge as input information. By incorporating input distributions and prior knowledge, the FIFK operator can address the limitations of using high-dimensional input distributions by semi-supervised assumptions, leading to more precise approximations of the integral operator. Additionally, the FIFK's incorporation of probabilistic principles can further enhance the accuracy and effectiveness of solutions. In cases of noisy operator equations and limited data, the FIFK's flexibility in defining problems using prior information or cross-validation with various kernel designs is especially advantageous. This capability allows for detailed problem definitions and facilitates achieving high levels of accuracy and stability in solutions. In our study, we examined the FIFK through two different approaches. Firstly, we implemented a semi-supervised assumption by using the same Fredholm operator kernel and data function kernel and incorporating unlabeled information. Secondly, we used the MSDF method, which involves selecting different kernels on both sides of the equation to define when the mapping kernel is different from the data function kernel. To assess the effectiveness of the FIFK and the proposed methods in solving ill-posed problems, we conducted experiments on a real-world dataset. Our goal was to compare the performance of these methods against the widely used least-squares method and other comparable methods.
BibTeX
@misc{khanjari2023manually,
  author = {Amir Khanjari and Saeid Pourmand and Mohammad Reza Faridrohani},
  title = {{Manually Selecting The Data Function for Supervised Learning of small datasets}},
  year = {2023},
  eprint = {2303.03894},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.03894},
}
Figure