Nucleus Neural Network: A Data-driven Self-organized Architecture

Jia Liu, Wenhua Zhang, Maoguo Gong, Haibo HeIJCNN 2020 · OpenReview

TL;DR

This paper introduces the Nucleus Neural Network (NNN), a novel architecture that abandons traditional layered structures for a more flexible, self-organizing network inspired by biological brain nuclei. NNN excels at learning robust representations, particularly in scenarios with varying or irrelevant background information, by prioritizing connections based on data relevance.

What Problem Does This Paper Address?

Traditional deep neural networks rely on hierarchical, layered architectures. While successful, these rigid structures can be suboptimal for tasks where information flow is not strictly sequential or where irrelevant features (like background noise) can significantly degrade performance. Existing architecture search methods are computationally expensive, often requiring extensive training to evaluate potential network designs. This paper aims to address these limitations by proposing a more flexible architecture and a computationally efficient method for learning both the network's structure and its parameters, with a specific focus on robustness to variations in input data.

Methods

The core innovation is the Nucleus Neural Network (NNN). Unlike traditional networks, NNNs consist of an input layer, an output layer, and a "nucleus" where neurons can connect to any other neuron without strict layer constraints. Feedback connections are disallowed for time-independent data.

The architecture learning process is data-driven and prioritizes connecting density based on the relevance between input and output neurons. This relevance is quantified using normalized pointwise mutual information (NPMI). The objective is to maximize the probability of observed data while minimizing the probability of all other data, achieved through a Products of Experts (PoE) model. To overcome the computational burden of traditional architecture search, they propose an objective function that models connections without involving network weights and biases. This objective is then optimized using a Binary Particle Swarm Optimization (BPSO) algorithm.

For parameter learning (weights and biases), they introduce an error-driven probability model (EDPM). This model, inspired by Restricted Boltzmann Machines (RBMs), constructs a probability density that captures observed data distribution given a reference output. Optimization involves a novel gradient calculation that uses both observed data and data sampled from the network's current distribution to mitigate the uneven impact of errors from different path depths.

Datasets

  • MNIST Dataset: Used for initial training and evaluation on a standard classification task.
  • Reconstructed Dataset: A new dataset created by combining MNIST digits with diverse backgrounds (random image patches and random noise). This dataset is designed to challenge traditional learners by having independent training and test sets with different distributions, specifically to test NNN's robustness to irrelevant components.

Key Insights and Ideas

  • Flexibility over Rigidity: Abandoning strict layers allows for more adaptive and potentially more powerful information flow, mimicking the complex connectivity in biological brains.
  • Data-Driven Connectivity: Prioritizing connections based on the statistical relevance between input and output features leads to more efficient and meaningful network structures.
  • Computational Efficiency: Modeling architecture learning independently of weights and biases significantly reduces the computational cost of finding optimal network designs.
  • Robustness to Irrelevant Information: NNNs demonstrate a remarkable ability to ignore or downplay irrelevant features (like background noise), leading to improved performance on datasets with varying or corrupted inputs. This is a significant advantage over traditional architectures that struggle with such variations.

Key Results

  • On the standard MNIST dataset, NNN's performance was comparable to traditional methods like DBN and CNN, though CNN achieved slightly higher accuracy.
  • On the reconstructed dataset with corrupted backgrounds (bg-img and bg-rand), NNN significantly outperformed traditional architectures (CNN, DBN) in terms of AUC, AP, and CA. For instance, on the bg-rand test set, NNN with EDPM achieved an AUC of 0.8926 and CA of 67.00%, far surpassing CNN and DBN.
  • NNN achieved this performance with significantly fewer processing units and connections compared to DBN and CNN.

Limitations and Future Work

  • Learning Efficiency: The authors acknowledge that the architecture and parameter learning efficiency of NNN still has room for improvement.
  • Classification Accuracy: While robust, NNN's classification accuracy on the standard MNIST task did not surpass that of CNN.
  • Future Work: The authors plan to further enhance learning efficiency, explore feedback connections, and stack NNNs to mimic hierarchical architectures found in the brain.

Summary updated

Abstract

In this paper, inspired from the nuclei in brain, we propose a nucleus neural network (NNN) and corresponding connecting architecture learning method. In a nucleus, the neurons are not assigned as regular layers, i.e., a neuron may connect to any neurons in the nucleus and the connections are self-organized according to data distribution. This type of architecture gets rid of layer limitation and makes full use of processing capability of each neuron. It is crucial to assign connections between all the neuron pairs. To address the time demanding of objectives in traditional architecture learning methods, we propose an efficient architecture learning model for the nucleus based on the principle that more relevant input and output neuron pair deserves higher connecting density. The new objective measures the information flow through the network architecture without involvement of weights and biases which greatly reduces the computational complexity. We find that this novel architecture is robust to irrelevant components in test data. So we reconstruct a new dataset based on the MNIST dataset where the types of digital backgrounds in training and test sets are different. The new dataset is a great challenge for learners because training data and test data are not only independent but also follow different distributions. Experiments demonstrate that NNN achieves significant improvement over architectures with regular layers on the reconstructed dataset.
BibTeX
@inproceedings{liu2020nucleus,
  author = {Jia Liu and Wenhua Zhang and Maoguo Gong and Haibo He},
  title = {{Nucleus Neural Network: A Data-driven Self-organized Architecture}},
  booktitle = {IJCNN},
  year = {2020},
  url = {https://openreview.net/forum?id=69UpcITyqH},
}
Figure