Automatic Discoveries of Physical and Semantic Concepts via Association Priors of Neuron Groups

Shuai Li, Kui Jia, Xiaogang WangarXiv 2016 · arXiv

Introduction

Deep neural networks are highly effective at image classification, yet their internal representations often remain a "black box." Standard supervised learning provides guidance on final outputs but offers little control over how intermediate layers represent the world. This paper introduces GSMax (Group Softmax), a novel association prior designed to bridge this gap by encouraging networks to automatically organize internal neurons into meaningful, real-world concepts.

Methodology: Group Softmax (GSMax)

The central contribution is an architectural shift from treating neurons as independent units to organizing them into small, competitive groups. This structure is grounded in mathematical group theory, where a "concept" is defined as an orbit—a collection of variations of a prototype (such as an edge at different rotations).

Architectural comparison: The GSMax approach (left) groups neurons to enforce competition and specialization, unlike standard fully connected layers (right).

By applying the GSMax activation function, the network enforces a "winner-takes-all" dynamic within each group. This competition forces specialization: for a given input, only one neuron in a group is active, incentivizing each neuron to represent a distinct "variation" or "sub-concept" of a larger category. This "Competitive Group-restricted Boltzmann Machine" approach simplifies complex symmetry discovery into a practical activation function.

Experimental Evaluation

The researchers evaluated GSMax across three benchmarks: CIFAR-10, CIFAR-100, and a custom "Small ImageNet." The most critical test involved "super-class" training. For instance, on CIFAR-100, the network was trained only on 20 super-class labels (e.g., "Aquatic Mammals") to determine if it could autonomously discover the 100 underlying sub-classes (e.g., "Whale," "Dolphin") without explicit supervision.

Key Results

The quantitative results demonstrate that the GSMax prior significantly outperforms standard architectures in discovering hidden sub-classes. By preventing multiple neurons in a group from firing simultaneously, the network is forced to delegate specific "responsibilities" to each neuron, leading to implicit semantic learning.

Quantitative results for sub-class discovery on CIFAR-100 and Small ImageNet benchmarks.

As shown in the table above, the GSMax-equipped network ("Implicit Learned") achieved 43.96% accuracy on CIFAR-100 sub-class discovery. In contrast, a standard control group architecture only reached 22.47%, which is marginally above the 20.00% chance level. This performance gap highlights the effectiveness of the association prior in guiding the network toward a hierarchical understanding of data, approaching the performance of architectures with much higher levels of supervision.

Conclusion and Future Work

The GSMax prior demonstrates that simple architectural constraints can lead to the discovery of complex semantic and physical hierarchies. While the current approach requires manual definition of group sizes, future research aims to develop adaptive grouping methods where the network dynamically determines the number and size of concept groups based on the data. Moving beyond simple physical symmetries to complex transformations like 3D viewpoint changes remains a promising frontier for making deep learning more interpretable and structured.

Summary updated

Abstract

The recent successful deep neural networks are largely trained in a supervised manner. It {\it associates} complex patterns of input samples with neurons in the last layer, which form representations of {\it concepts}. In spite of their successes, the properties of complex patterns associated a learned concept remain elusive. In this work, by analyzing how neurons are associated with concepts in supervised networks, we hypothesize that with proper priors to regulate learning, neural networks can automatically associate neurons in the intermediate layers with concepts that are aligned with real world concepts, when trained only with labels that associate concepts with top level neurons, which is a plausible way for unsupervised learning. We develop a prior to verify the hypothesis and experimentally find the proposed prior help neural networks automatically learn both basic physical concepts at the lower layers, e.g., rotation of filters, and highly semantic concepts at the higher layers, e.g., fine-grained categories of an entry-level category.
BibTeX
@misc{li2016automatic,
  author = {Shuai Li and Kui Jia and Xiaogang Wang},
  title = {{Automatic Discoveries of Physical and Semantic Concepts via Association Priors of Neuron Groups}},
  year = {2016},
  eprint = {1612.09438},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1612.09438},
}
Figure