Compositional Deep Learning

Bruno GavranovicarXiv 2019 · arXiv

Abstract

This paper presents a formal mathematical framework for deep learning based on Category Theory, as developed in the Master’s thesis by Bruno Gavranović (2019). By treating neural network architectures as functors and parameters through the Para construction, the research moves optimization from a search in function space to a search in functor space. This categorical approach provides a rigorous language for describing structural invariants in neural systems and introduces the "Product Task"—a novel method for unpaired image-to-image translation that treats images as categorical products of their constituent components. Experimental results on the CelebA dataset and StyleGAN latent spaces demonstrate the viability of using categorical principles to perform complex object manipulation tasks.

Introduction

Despite the empirical success of deep learning, the design of neural architectures remains largely an intuitive and ad-hoc process. There is a lack of formal language to describe the structural rules—the invariants—that remain fixed even as the weights of a model change during training. This research addresses this gap by bridging the abstract world of Category Theory with the practical requirements of deep learning. By formalizing architectures as mathematical objects, we can derive new optimization strategies and architectural constraints that are grounded in rigorous logic rather than heuristic experimentation.

Theoretical Framework

The Para Construction and Parametrized Morphisms

To represent neural networks categorically, the author utilizes the Para construction. In standard category theory, a morphism is a function $f: X \to Y$. However, neural networks are parametrized; they represent a family of functions. The Para construction allows for "parametrized morphisms," where a network is defined as $f: P \times X \to Y$, where $P$ represents the parameter space. This allows the framework to treat the learning process as a search within a structured category of Euclidean spaces (Para(Euc)).

Architectures as Functors

A central contribution of this work is the definition of a "model schema"—a category that represents the structural blueprint of a network. A specific neural network is then viewed as a functor from this schema to the category of parametrized morphisms. This shift is significant: it implies that training a model is equivalent to finding a functor that satisfies specific integrity constraints defined by the architecture's geometry.

Path-Equivalence and Generalized Loss

The framework generalizes existing generative models, such as GANs and CycleGANs, into a unified "path-equivalence loss." In this categorical view, the cycle-consistency loss used in unpaired translation is a requirement that different paths through the categorical schema must be equivalent. For example, the composition of "removing an object" and "adding an object" should result in an identity morphism, returning the original image. This provides a mathematical foundation for why certain loss functions work and how they can be generalized to more complex topologies.

The Product Task

The author introduces the "Product Task" as a novel application of these principles. By modeling an image as a categorical product ($AB \cong A \times B$), the network learns the isomorphism between a composite image and its individual factors. This allows the model to perform object insertion and deletion (e.g., adding or removing eyeglasses) using unpaired data. The inductive bias—the prior knowledge that an image can be decomposed into independent components—is baked into the categorical structure of the model itself.

Experimental Evaluation

The framework was tested across three distinct domains to verify its theoretical claims:

  1. Synthetic Circles: A toy dataset of unicolored circles on striped backgrounds was used to prove that the model could successfully learn to separate and recompose independent visual factors.
  2. CelebA Faces: Using the "Eyeglasses" attribute, the model was tasked with unpaired image translation. It successfully learned to remove and insert various styles of glasses onto celebrity faces, demonstrating that the categorical approach scales to high-dimensional, real-world data.
  3. StyleGAN Latent Space: The author explored whether the Product Task could operate within the 512-dimensional latent space of a pre-trained StyleGAN. This experiment treated the latent vectors themselves as categorical objects, attempting to find directions in the latent space that correspond to specific object manipulations.

Results and Discussion

The research successfully demonstrated that standard generative architectures are specific instances of a broader categorical framework. On the CelebA dataset, the model achieved impressive results in object manipulation using only unpaired data. While the experiments in the StyleGAN latent space were more challenging to optimize—often struggling to balance adversarial and path-equivalence losses—they proved the theoretical possibility of performing categorical operations on learned representations.

Conclusion

This work establishes that Category Theory is not merely an abstract tool for classification but a viable framework for designing and training deep learning systems. By defining architectures as functors and tasks as categorical constructions, researchers can introduce rigorous inductive biases into their models. Future work remains in extending this framework to dynamic architectures like Recurrent Neural Networks and improving the optimization stability of categorical losses in complex latent spaces.

Summary updated

Abstract

Neural networks have become an increasingly popular tool for solving many real-world problems. They are a general framework for differentiable optimization which includes many other machine learning approaches as special cases. In this thesis we build a category-theoretic formalism around a class of neural networks exemplified by CycleGAN. CycleGAN is a collection of neural networks, closed under composition, whose inductive bias is increased by enforcing composition invariants, i.e. cycle-consistencies. Inspired by Functorial Data Migration, we specify the interconnection of these networks using a categorical schema, and network instances as set-valued functors on this schema. We also frame neural network architectures, datasets, models, and a number of other concepts in a categorical setting and thus show a special class of functors, rather than functions, can be learned using gradient descent. We use the category-theoretic framework to conceive a novel neural network architecture whose goal is to learn the task of object insertion and object deletion in images with unpaired data. We test the architecture on three different datasets and obtain promising results.
BibTeX
@misc{gavranovic2019compositional,
  author = {Bruno Gavranovic},
  title = {{Compositional Deep Learning}},
  year = {2019},
  eprint = {1907.08292},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.08292},
}
Figure