Abstract
We propose a new approach, called cooperative neural networks (CoNN), which uses a set of cooperatively trained neural networks to capture latent representations that exploit prior given independence structure. The model is more flexible than traditional graphical models based on exponential family distributions, but incorporates more domain specific prior structure than traditional deep networks or variational autoencoders. The framework is very general and can be used to exploit the independence structure of any graphical model. We illustrate the technique by showing that we can transfer the independence structure of the popular Latent Dirichlet Allocation (LDA) model to a cooperative neural network, CoNN-sLDA. Empirical evaluation of CoNN-sLDA on supervised text classification tasks demonstrates that the theoretical advantages of prior independence structure can be realized in practice—we demonstrate a 23% reduction in error on the challenging MultiSent data set compared to state-of-the-art.
Introduction: The Gap Between Deep Learning and PGMs
Deep neural networks (DNNs) have revolutionized machine learning through their ability to learn complex, non-linear representations from raw data. However, DNNs typically operate as "blank slates," requiring vast amounts of data to learn domain-specific relationships that might be known a priori. Conversely, Probabilistic Graphical Models (PGMs), such as Latent Dirichlet Allocation (LDA), excel at encoding human knowledge through explicit independence structures. Yet, PGMs are often mathematically rigid, relying on restrictive probability distributions (like the exponential family) to maintain tractable inference.
This research introduces Cooperative Neural Networks (CoNN), a framework designed to bridge this gap. CoNN explicitly encodes the structural priors of a graphical model while leveraging the flexible, non-linear learning capabilities of neural networks.
Methodology: Cooperative Neural Networks
The core idea of CoNN is to map the independence structure of a PGM into a Hilbert space. Instead of solving variational inference equations using fixed distributional forms, the authors use embeddings to represent variables in a potentially infinite-dimensional feature space. This preserves the relationships between variables without the need for conjugate priors.
The complex mathematical operators of the Hilbert space are then parameterized by neural networks. These networks are trained "cooperatively," meaning they iteratively update their embeddings to remain consistent with each other and the observed data, optimized end-to-end via backpropagation and a classification loss function.
Implementation: CoNN-sLDA
The authors demonstrate this framework by adapting the structure of supervised LDA (sLDA) into CoNN-sLDA. This model factors local word-level information and global document-level information into separate components, mirroring the generative process of LDA but using expressive neural updates.
As shown in the architecture diagram, the model uses word embeddings (e.g., word2vec) as inputs. It employs shallow neural networks (typically single-layer with tanh activations) to compute the latent representations mu_z and mu_theta. Because the architecture is guided by the underlying graphical structure, it achieves high performance without the need for ultra-deep layers.
Experimental Results
CoNN-sLDA was evaluated on two benchmark text classification tasks: MultiSent (342k Amazon reviews) and 20 Newsgroups.
- MultiSent: CoNN-sLDA achieved an AUC of 93.4%, outperforming the previous state-of-the-art deep topic model (DUI-sLDA, 91.4%). This represents a significant 23% reduction in error rate.
- 20 Newsgroups: The model achieved 83.7% accuracy, surpassing traditional variational inference (73.8%) and other deep learning baselines like BP-sLDA (81.8%).
The t-SNE visualization demonstrates that the model learns highly discriminative document embeddings that cluster effectively by topic, even with a relatively shallow architecture. Furthermore, the model is computationally efficient; training on the large MultiSent dataset takes approximately five minutes on a single NVIDIA Tesla P100 GPU.
Discussion and Future Work
The CoNN framework offers a powerful way to inject structural priors into deep learning. While the current implementation focuses on mean-field inference, the authors suggest extending it to other techniques like loopy belief propagation. A key limitation noted is the sensitivity to embedding dimensionality and network depth, where over-parameterization can lead to overfitting. Future work also includes improving the interpretability of the learned Hilbert space embeddings by mapping them back to the original word space to recover topic-word distributions.
Summary updated