ConformalLayers: A non-linear sequential neural network with associative layers

Eduardo Vera Sousa, Leandro A. F. Fernandes, Cristina Nader Vasconcelos

Introduction

In modern Deep Learning, the pursuit of higher accuracy has traditionally led to increasingly deep neural network architectures. However, this depth comes at a significant computational cost: every additional layer increases the number of floating-point operations and the memory footprint required to store intermediate feature maps. This paper introduces ConformalLayers, a novel architectural paradigm that breaks this linear relationship between depth and inference cost.

The Challenge of Non-Linearity

The primary obstacle to simplifying deep networks is the non-linear activation function (e.g., ReLU, Sigmoid). While a sequence of linear layers (like convolutions) can be mathematically collapsed into a single operation, the interleaving of non-linearities prevents this "associativity." Consequently, standard networks must compute each layer sequentially, storing the output of one to serve as the input for the next.

Methods: The ReSPro Activation

To overcome this, the authors leverage Conformal Geometry to design a new, associative non-linear activation function called ReSPro (Spherical Reflection, Scaling, and Projection). By mapping input data into a higher-dimensional conformal space, ReSPro acts as a non-linear transformation that can be represented as a sequence of linear operations within that specific geometric domain.

Geometric representation of the ReSPro activation mapping points into a conformal space.

Because ReSPro and standard operations (convolutions, pooling, dropout) are all associative within this conformal model, an entire sequence of layers can be treated as a single chain of operations. Following training, the authors apply the associative property $(A \circ B) \circ C = A \circ (B \circ C)$ to collapse the network into two static components: a sparse matrix ($L_M$) and a sparse rank-3 tensor ($L_T$). At runtime, the network performs a single calculation: $Y = (L_M + L_T X)X$. This ensures that whether the original design had 5 layers or 500, the inference time remains constant.

Key Results and Performance

The researchers evaluated ConformalLayers using standard benchmarks including MNIST, Fashion-MNIST, and CIFAR-10. The results highlight a transformative shift in throughput and memory efficiency.

Inference Speed and Throughput

In tests comparing varying depths, ConformalLayers demonstrated depth-independent inference time, becoming more efficient than traditional CNNs once the network exceeded 9 layers. The most significant advantage, however, appeared in batch processing capacity.

Inference time comparison showing the superior batch capacity of ConformalLayers (D3ModNetCL) compared to standard models.

As shown in Figure 1, the proposed method (D3ModNetCL) scales linearly with batch size far beyond the limits of traditional architectures. While a standard network crashed after 14,000 images on an NVIDIA GTX 1050 Ti due to memory constraints, ConformalLayers successfully processed 89,000 images simultaneously, achieving roughly 1.16x faster inference at maximum capacity.

Accuracy Trade-offs

This efficiency comes with a small "associativity tax." On MNIST, ConformalLeNet achieved 97.33% accuracy (vs. 98.87% for standard LeNet). The gap widened on CIFAR-10 (53.27% vs. 59.05%), indicating that while ReSPro is a powerful global activation, it currently lacks the full expressive power of ReLU for highly complex patterns.

Discussion and Future Work

ConformalLayers represents a significant proof-of-concept for constant-cost deep learning. While current limitations include a lack of support for bias terms and non-sequential graphs (like ResNets), the ability to collapse depth into a single operation offers a new path for deploying deep models on resource-constrained hardware. Future research will focus on narrowing the accuracy gap and extending associativity to complex network topologies via depth-first graph traversal.

Summary updated

Abstract

Convolutional Neural Networks (CNNs) have been widely applied. But as the CNNs grow, the number of arithmetic operations and memory footprint also increase. Furthermore, typical non-linear activation functions do not allow associativity of the operations encoded by consecutive layers, preventing the simplification of intermediate steps by combining them. We present a new activation function that allows associativity between sequential layers of CNNs. Even though our activation function is non-linear, it can be represented by a sequence of linear operations in the conformal model for Euclidean geometry. In this domain, operations like, but not limited to, convolution, average pooling, and dropout remain linear. We take advantage of associativity to combine all the “conformal layers” and make the cost of inference constant regardless of the depth of the network.
BibTeX
@inproceedings{sousa2021conformallayers,
  author = {Eduardo Vera Sousa and Leandro A. F. Fernandes and Cristina Nader Vasconcelos},
  title = {{ConformalLayers: A non-linear sequential neural network with associative layers}},
  booktitle = {SIBGRAPI},
  year = {2021},
  url = {https://openreview.net/forum?id=R49KVZhiIIn},
}
Figure