Representation Learning in Fully Connected Neural Networks

56 concepts56 papers2 proposals

Linear models are fundamentally limited by their geometry: they can only separate data using straight lines, planes, or hyperplanes. When real-world phenomena involve intricate feature interactions or thresholds, linear transformations fail. The Multi-Layer Perceptron (MLP), or feedforward neural network, overcomes this limitation by stacking layers of artificial neurons combined with non-linear activation functions. This architecture allows the network to warp, bend, and transform the input space into a representation where complex relationships become linearly predictable.

The Geometry of Layered Abstraction

At its core, a fully connected feedforward network passes an input vector through a sequence of hidden layers to produce an output. Each layer computes a weighted sum of its inputs and adds a bias term, which can be expressed mathematically as a linear transformation:

$$z = Wx + b$$

If a network consisted solely of these linear transformations, stacking multiple layers would provide no benefit. A composition of linear functions is simply another linear function; a ten-layer linear network has no more representational capacity than a single-layer linear model.

To break this linearity, feedforward networks apply an element-wise non-linear activation function—such as the Rectified Linear Unit (ReLU), Sigmoid, or Exponential Linear Unit (ELU)—to the output of each hidden layer. This architectural choice enables the network to learn complex, non-linear mappings. As early applications demonstrated, such as the seismic analysis detailed in Lg Depth Estimation and Ripple Fire Characterization Using Artificial Neural Networks, passing inputs through successive layers of nodes with sigmoidal transformations allows the network to capture physical insights that escape simple linear discrimination.

Why Depth Beats Breadth

The Universal Approximation Theorem states that a network with a single hidden layer can approximate any continuous function, provided it has enough neurons. However, a shallow network may require an exponentially large number of neurons, making it computationally impractical and prone to memorization rather than generalization.

Increasing the depth of the network introduces a hierarchical prior. Instead of learning a flat mapping, a deep feedforward network decomposes a complex function into a series of simpler, nested functions. As Ebru Arisoy et al. demonstrated in Deep Neural Network Language Models, stacking multiple hidden layers with non-linear activations allows a model to learn hierarchical, abstract relationships between input features. Crucially, this architectural shift improves performance even when the total parameter budget is kept constant relative to shallower models.

Input Layer  ──>  Hidden Layer 1  ──>  Hidden Layer 2  ──>  Output Layer
 (Features)        (Low-level          (High-level          (Prediction)
                  Abstractions)       Abstractions)

Each subsequent layer composes the features learned by the previous layer. For instance, in remote sensing applications like the one explored by Lu She et al. in Himawari-8 Aerosol Optical Depth (AOD) Retrieval Using a Deep Neural Network Trained Using AERONET Observations, a multi-layer architecture with progressively increasing hidden layer widths (256, 512, and 512 neurons) allows the network to systematically combine raw spectral reflectances and auxiliary data into highly accurate physical predictions.

Mapping High-Dimensional Tabular Data

Fully connected networks are uniquely suited for heterogeneous tabular data where spatial or temporal grid structures—like those exploited by convolutional or recurrent networks—are absent. In fields like clinical diagnostics, input features are often a disparate mix of categorical variables, demographic data, and continuous laboratory measurements.

The dense connectivity of an MLP ensures that every input feature has the opportunity to interact with every other feature. The authors of Prognostic Modeling of COVID-19 Using Artificial Intelligence in the United Kingdom: Model Development and Validation leveraged this capability by optimizing a multi-layer network with three densely connected hidden layers. This structure was designed to extract and combine high-level patterns from basic clinical features available at the moment of hospital admission, capturing non-linear risk factors that shallow statistical methods might miss.

Similarly, Shuyan Cai et al. in Fully-connected network-based prediction model for lymph node metastasis in clinical early-stage endometrial cancer: development and validation in two centers utilized a multi-layer perceptron that progressively reduces dimensionality across hidden layers to map a wide array of preoperative clinical variables to a binary classification output.

Despite their power, the high capacity of fully connected networks requires careful justification. In a comprehensive systematic review, Jessica Doohan et al. noted in Comparison of generalised additive models and neural networks in applications: A systematic review that while MLPs serve as an essential high-capacity baseline for tabular datasets, simpler statistical models like Generalised Additive Models (GAMs) often remain highly competitive while offering superior interpretability.

Managing Capacity and Overfitting

Because fully connected layers contain a weight parameter for every single connection between adjacent layers, they are highly susceptible to overfitting, particularly when training data is scarce or highly dimensional. If unregularized, the network will simply memorize noise in the training set rather than learning the underlying function.

To mitigate this, modern feedforward architectures rely on specific structural motifs:

  • Information Bottlenecks: Designing networks with decreasing layer widths forces the model to compress information, retaining only the most salient features as data flows deeper.
  • Dropout: Randomly deactivating a percentage of neurons during each training step prevents neurons from co-adapting, forcing the network to learn redundant, robust representations.

For example, Yangyang Miao et al. in Machine Learning-based Prediction Analysis of Potential Factors in Traffic Accidents highlighted the effectiveness of an architecture utilizing multiple dense layers of decreasing width interspersed with dropout layers to capture complex interactions without overfitting.

A similar strategy was employed by Yuzhen Huang et al. in Constructing a prediction model for lymph node metastasis in patients with incidental finding of endometrial cancer based on Fully-Connected Network, where a multi-layer perceptron combined decreasing hidden layer widths with a high dropout rate to safely map high-dimensional clinical variables to reliable binary outcomes. By balancing depth, non-linear activation, and regularization, feedforward architectures transform raw, unorganized data into structured, hierarchical representations.

Key Foundational Work

Lg Depth Estimation and Ripple Fire Characterization Using Artificial Neural Networks (1990) — contributed 1 concept elements to this direction.
Part-of-Speech Tagging With Neural Networks (1994) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_deep_neural_network_architecture_d8bfe19c,
  title = {deep neural network architecture},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/deep-neural-network-architecture/rdr_d8bfe19cb70045aeba1404e602b18afd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_deep_neural_network_architecture_d8bfe19c_v1,
  title = {deep neural network architecture},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/deep-neural-network-architecture/rdr_d8bfe19cb70045aeba1404e602b18afd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake