Nucleus Neural Network for Super Robust Learning

Jia Liu, Maoguo Gong, Haibo HearXiv 2019 · OpenReview

TL;DR

This paper introduces the Nucleus Neural Network (NNN), a novel architecture that removes traditional layer limitations, allowing neurons to connect freely within a "nucleus." This data-driven approach learns connections based on input-output relevance, leading to a network that is remarkably robust to irrelevant input components, particularly background noise in image classification.

What Problem Does This Paper Address?

Traditional deep neural networks rely on hierarchical, layered structures. While successful, these architectures can be rigid and struggle when test data differs significantly from training data, especially in the presence of irrelevant features like background variations. The paper aims to overcome this limitation by proposing a more flexible architecture that can learn to prioritize relevant information and ignore distractions, leading to improved robustness and generalization.

Methods

The core innovation is the Nucleus Neural Network (NNN) architecture. Instead of fixed layers, NNN features a "nucleus" where neurons can connect to any other neuron. The architecture learning process is data-driven:

  • Connecting Density: A principle is established where more relevant input-output neuron pairs should have higher connection density. This density is propagated through the network.
  • Objective Function: An efficient objective function is proposed to model the architecture. It leverages the concept of "products of experts" (PoE) to model data distributions and uses normalized pointwise mutual information (NPMI) to quantify relevance between input and output neurons. This approach allows architecture optimization without needing to train weights and biases, significantly reducing computational cost.
  • Optimization: A binary particle swarm optimization (BPSO) algorithm is used to optimize the architecture based on the derived objective function.
  • Parameter Learning: After the architecture is optimized, weights and biases are learned using a novel error-driven probability model (EDPM), which is inspired by Restricted Boltzmann Machines (RBMs) and aims to capture the distribution of observed data.

Datasets

  • MNIST: The standard MNIST dataset is used for initial training and evaluation on a traditional classification task.
  • Reconstructed MNIST Dataset: A new dataset is created by augmenting the MNIST dataset with different background types (random image patches and random noise) for both training and testing. This dataset is specifically designed to test the robustness of NNN to irrelevant input components.

Key Insights and Ideas

  • Breaking Layer Limitations: The NNN architecture fundamentally challenges the necessity of strict layer-by-layer processing, allowing for more complex and potentially more efficient information flow.
  • Data-Driven Architecture: The architecture is not predefined but learned directly from the data's inherent relevance between inputs and outputs. This makes it adaptable to specific tasks.
  • Robustness to Irrelevant Features: The key insight is that by focusing on relevance, NNN naturally learns to downplay or route irrelevant information (like background) through less impactful paths, leading to significant robustness.
  • Efficient Architecture Search: By decoupling architecture optimization from weight training, the paper offers a computationally more feasible approach to neural network architecture search.

Key Results

  • Traditional Classification: On the standard MNIST dataset, NNN performs comparably to traditional methods like DBN and CNN, though CNN achieves slightly higher accuracy due to its specialized architecture.
  • Robustness on Reconstructed Dataset: On the newly constructed dataset with varied backgrounds, NNN significantly outperforms traditional DBN and CNN architectures. For instance, on the "bg-rand" test set, NNN achieves an AUC of 0.8926 with EDPM, compared to CNN's 0.7434 and DBN's 0.7082. This demonstrates its superior ability to handle unseen background variations.
  • Parameter Efficiency: NNN achieves competitive performance with fewer parameters and processing units compared to DBN and CNN.

Limitations and Future Work

  • Learning Efficiency: The authors acknowledge that the architecture and parameter learning efficiency still have room for improvement.
  • Classification Accuracy: While robust, the classification accuracy on the reconstructed dataset is lower than on the original MNIST dataset, indicating a trade-off or areas for further optimization.
  • Future Directions: The authors suggest further improvements in learning efficiency, the exploration of feedback connections, and stacking NNNs to mimic hierarchical architectures found in the brain.

Summary updated

Abstract

Artificial neural networks which are inspired from the learning mechanism of brain have achieved great successes in many problems, especially those with deep layers. In this paper, we propose a nucleus neural network (NNN) and corresponding connecting architecture learning method. In a nucleus, there are no regular layers, i.e., a neuron may connect to all the neurons in the nucleus. This type of architecture gets rid of layer limitation and may lead to more powerful learning capability. It is crucial to determine the connections between them given numerous neurons. Based on the principle that more relevant input and output neuron pair deserves higher connecting density, we propose an efficient architecture learning model for the nucleus. Moreover, we improve the learning method for connecting weights and biases given the optimized architecture. We find that this novel architecture is robust to irrelevant components in test data. So we reconstruct a new dataset based on the MNIST dataset where the types of digital backgrounds in training and test sets are different. Experiments demonstrate that the proposed learner achieves significant improvement over traditional learners on the reconstructed data set.
BibTeX
@misc{liu2019nucleus,
  author = {Jia Liu and Maoguo Gong and Haibo He},
  title = {{Nucleus Neural Network for Super Robust Learning}},
  year = {2019},
  eprint = {1904.04036},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1904.04036},
}
Figure