Introduction
Despite the widespread adoption of deep learning, the design of neural network architectures remains largely empirical. Layers and kernels are often added based on heuristic success rather than rigorous mathematical foundations. A significant challenge in current architectures is redundancy: when multiple neurons in a hidden layer learn identical features, the network becomes inefficient and its generalization error increases. This paper introduces the Minimize Mutual Information Method (MMIM), a regularization technique designed to reduce redundancy and improve interpretability by minimizing a theoretical upper bound on generalization error.
Methodology
MMIM is grounded in information theory, specifically targeting the decomposition of a generalization error upper bound, denoted as $g$. This bound is partitioned into three primary components:
- $C_1$: Represents the redundancy and diversity of hidden units.
- $C_2$: Represents the entropy of labels (constant for a specific dataset).
- $C_3$: Represents the mutual information between network weights and hidden representations.
The authors propose a two-step optimization process. First, $C_1$ is minimized to ensure that hidden neurons learn diverse, non-overlapping features. Second, the parameters of the initial layers are fixed while $C_3$ is minimized to further tighten the error bound. To overcome the difficulty of calculating mutual information in high-dimensional spaces, the method integrates Mutual Information Neural Estimation (MINE), allowing for gradient-based optimization via backpropagation.
Experimental Setup
The efficacy of MMIM was evaluated using the Fashion-MNIST dataset, which contains 70,000 grayscale images across 10 clothing categories. The method was tested across two standard architectures:
- Deep Neural Networks (DNN): Fully connected layers.
- Convolutional Neural Networks (CNN): Standard convolutional layers.
Results and Discussion
The experimental results demonstrate that MMIM consistently improves classification accuracy compared to baseline models and models using only partial components of the regularization bound ($C_4, C_5, C_6$).
As shown in Table 1, the full MMIM implementation achieved the highest accuracy for both DNN (90.82%) and CNN (92.39%) architectures. By treating neurons as an ensemble of independent "base learners," MMIM forces individual units to specialize in distinct features—such as specific textures or contours—rather than learning redundant information. The results confirm that minimizing the holistic theoretical bound is more effective than optimizing individual components like redundancy ($C_4$) alone.
Conclusion
MMIM provides a mathematically grounded approach to regularization that enhances both performance and interpretability. By reducing redundancy, it creates more efficient networks with tighter generalization bounds. While currently tested on image classification benchmarks, future work will explore the application of MMIM to larger datasets and its potential as a blueprint for designing novel, non-redundant network architectures from the ground up.
Summary updated