LMKL-Net: A Fast Localized Multiple Kernel Learning Solver via Deep Neural Networks

Ziming ZhangarXiv 2018 · OpenReview

TL;DR

This paper introduces LMKL-Net, a novel deep neural network architecture that efficiently solves Localized Multiple Kernel Learning (LMKL) problems. By using an attentional network for gating and a multilayer perceptron for classification, LMKL-Net learns both kernel weights and the decision function end-to-end, achieving state-of-the-art accuracy while being significantly faster and more memory-efficient than existing methods.

What Problem Does This Paper Address?

Multiple Kernel Learning (MKL) is a powerful technique for combining information from various sources (kernels) to improve machine learning model performance. A specific variant, Localized Multiple Kernel Learning (LMKL), aims to learn kernel weights that are dependent on the input data itself, allowing for more adaptive and context-aware kernel combinations. However, traditional LMKL methods face challenges: learning the "gating function" (which determines these localized weights) often requires manual tuning or strong prior assumptions, and optimizing these models can be computationally expensive and memory-intensive, especially for large datasets. This paper addresses the need for an automated, efficient, and accurate LMKL solver that can learn both the gating and classification functions without relying on hand-crafted priors.

Methods

The core innovation is LMKL-Net, a feedforward deep neural network designed to tackle LMKL. Instead of relying on traditional optimization algorithms, LMKL-Net treats the LMKL problem as a function approximation task.

  • Parameterizing Gating and Classification: The key idea is to use neural networks to learn the components of LMKL.
  • An Attentional Network (AN) is used to learn the gating function, which determines the localized weights for each kernel based on the input data. This replaces manual design of gating functions.
  • A Multilayer Perceptron (MLP) is used to learn the multiclass classifier.
  • End-to-End Learning: The AN and MLP are integrated into a single network. By sequentially applying the AN, an average pooling layer (to aggregate kernel information), and the MLP, the network learns to approximate the nonlinear decision function of LMKL.
  • Shared Weights for Efficiency: To further accelerate training and inference, the paper proposes sharing weights between the initial fully connected layers of the AN and the MLP, leading to a more compact architecture.
  • Optimization: The network is trained using the ADAM optimizer, a variant of stochastic gradient descent (SGD), which allows for efficient learning on large datasets.

Datasets

The paper evaluates LMKL-Net on a variety of benchmark datasets for both binary and multiclass classification tasks.

  • Binary Classification: adult-8, news20, phishing, rcv1, real-sim, w7a.
  • Multiclass Classification: aloi, covtype, letter, protein, sensit, sensorless, shuttle, SVHN.
  • Small-scale Dataset: VGG MKL dataset (based on Caltech-101).

The authors mention that for large-scale datasets, they intentionally sample a subset of data (20K for training/validation, 10K for testing) to manage memory and computational constraints, and precompute kernels using 10 Gaussian RBF kernels.

Key Insights and Ideas

  • Neural Networks as LMKL Solvers: The paper demonstrates that deep neural networks can effectively serve as direct solvers for LMKL, bypassing traditional, often complex, optimization procedures.
  • Learning Gating Functions: Using an attentional network to learn the gating function is a significant departure from prior work, allowing for data-driven, adaptive kernel weighting without explicit priors.
  • End-to-End Optimization: Training the gating and classification components jointly via SGD is more efficient than the alternating optimization strategies common in MKL.
  • Efficiency Gains: The network-based approach leads to dramatic improvements in training speed (orders of magnitude faster) and memory footprint, making LMKL more scalable.
  • Robustness: Despite potential overfitting observed on some datasets, LMKL-Net shows robustness in its ability to achieve good performance, suggesting the learned functions are well-behaved.

Key Results

  • Accuracy: LMKL-Net outperforms state-of-the-art MKL solvers on several benchmark datasets for both binary and multiclass classification. For binary classification, it achieved an average improvement of 8.49% over LMKL. For multiclass, it showed a 1.11% improvement over OBSCURE.
  • Speed: LMKL-Net is trained approximately two orders of magnitude faster than competing methods like OBSCURE, especially on large-scale datasets. For example, it takes about 6 seconds per epoch on a GPU compared to over 10 hours for OBSCURE on a CPU for a large dataset.
  • Memory Footprint: The active memory requirement for LMKL-Net is significantly smaller, estimated to be around 0.013 times that of loading precomputed kernels.
  • Learned Kernel Weights: The learned kernel weights are data-dependent and exhibit distinct patterns for different classes and datasets, validating the localized nature of the approach.

Limitations and Future Work

  • Convergence: The paper acknowledges that due to the non-convexity of the neural network, LMKL-Net achieves weak convergence in probability, a common characteristic of SGD-based optimization for deep networks.
  • Future Work: The authors suggest exploring extensions such as incorporating group sparsity into the network design for sparse learning in kernel SVMs and investigating other network architectures for different MKL learning problems. They also mention that further increasing the depth of the network doesn't necessarily improve accuracy and can increase computational burden.

Summary updated

Abstract

In this paper we propose solving localized multiple kernel learning (LMKL) using LMKL-Net, a feedforward deep neural network. In contrast to previous works, as a learning principle we propose {\em parameterizing} both the gating function for learning kernel combination weights and the multiclass classifier in LMKL using an attentional network (AN) and a multilayer perceptron (MLP), respectively. In this way we can learn the (nonlinear) decision function in LMKL (approximately) by sequential applications of AN and MLP. Empirically on benchmark datasets we demonstrate that overall LMKL-Net can not only outperform the state-of-the-art MKL solvers in terms of accuracy, but also be trained about {\em two orders of magnitude} faster with much smaller memory footprint for large-scale learning.
BibTeX
@misc{zhang2018lmkl,
  author = {Ziming Zhang},
  title = {{LMKL-Net: A Fast Localized Multiple Kernel Learning Solver via Deep Neural Networks}},
  year = {2018},
  eprint = {1805.08656},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1805.08656},
}

Loading related content...

Figure