In classical deep learning theory, a feedforward neural network is treated as a linear, sequential pipeline. Information enters the input layer and is processed step-by-step until it reaches the output. Mathematically, this structure is modeled as a Markov chain:
$$X \to H_1 \to H_2 \to \dots \to H_L \to Y$$
Under this view, each layer acts as a lossy filter. According to the Data Processing Inequality (DPI) from information theory, a Markov chain can only discard or reformat information; it can never create it. Therefore, the mutual information between the input $X$ and the hidden representations $H_l$ must strictly decrease or remain constant as depth increases:
$$I(X; H_1) \ge I(X; H_2) \ge \dots \ge I(X; H_L)$$
This elegant assumption underpins much of our traditional intuition about deep learning: deeper layers hold more abstract, highly refined, and less redundant representations.
However, modern architectures—such as ResNets, U-Nets, DenseNets, and Transformers—shatter this assumption. By introducing skip connections, residual blocks, and parallel branches, these networks allow information to bypass intermediate layers entirely. The strict Markovian sequence is replaced by a complex graph topology, giving rise to non-monotonic information flow.
The Breakdown of the Data Processing Inequality
When skip connections link non-adjacent layers, a deeper layer no longer depends solely on its immediate predecessor. For instance, if a skip connection feeds $H_1$ directly into $H_3$, bypassing $H_2$, the representation at $H_3$ can recover raw information that $H_2$ had discarded.
As Lee et al. demonstrate using information-theoretic analysis on U-Net architectures, this multi-stream routing explicitly violates simple Markovian assumptions. Because information can hop over layers, the classical Data Processing Inequality no longer applies.
Consequently, task-relevant information does not decrease monotonically with depth. Work by Soltani et al. reveals that deeper layers in skip-connection architectures can actually exhibit higher statistical dependency on the target labels than the shallower layers preceding them. Rather than a steady decay, information content fluctuates non-monotonically across the network's hierarchy.
Why Non-Monotonicity Breaks Standard Engineering Tools
This departure from Markovian dynamics is not just a theoretical curiosity; it breaks many of the standard heuristics used to analyze, optimize, and compress deep networks.
1. Network Pruning and Compression
Traditional structural pruning often relies on layer-wise ranking, assuming that deeper layers contain progressively more refined or redundant information. However, if information flows non-monotonically, a simple thresholding strategy might accidentally delete a deep layer that contains a vital surge of task-relevant information carried by a skip connection.
To solve this, Soltani et al. developed a framework called Pruning with Energy Dependence (PED). Instead of assuming a sequential hierarchy, they use model-free energy statistics to measure the direct dependency between each individual hidden layer and the final output, bypassing the network's internal topology to identify true redundancy. Similarly, Rostami et al. note that standard information-theoretic compression fails in modern architectures because channels become structurally "coupled" across parallel paths, requiring joint probability distributions to safely prune redundant blocks.
2. Uncertainty Calibration
When a network outputs a prediction, we want its confidence to match its true accuracy. One way to improve this "probability calibration" is to inspect intermediate layer embeddings to guide the final confidence score. Yet, as Zhang et al. show, the presence of skip connections disrupts this sequential flow. Because the network loses its Markovian properties, it becomes difficult to isolate which specific layers are responsible for uncertainty estimation, complicating the use of latent spaces for calibration.
3. Spectral and Symmetry Analysis
Theoretical tools designed to predict network performance at initialization often assume independent matrix products across layers. Chhaibi et al. explain that skip connections violate this independence, meaning standard spectral analysis tools fail.
Furthermore, proving how global properties propagate through a network becomes an open challenge. For example, Shahverdi et al. study how equivariance (e.g., rotation invariance) manifests internally. In a strictly feedforward network, global equivariance implies local, layerwise equivariance. But when skip connections introduce direct dependencies between non-adjacent layers, this clean, step-by-step propagation breaks down, making it mathematically difficult to prove that global functional constraints map to local constraints within individual layers.
New Frameworks for Non-Sequential Topologies
To accurately model and optimize networks with non-monotonic information flow, researchers are turning to alternative mathematical frameworks that abandon the sequential assumption entirely.
Sequential (Markovian): Input ──> Layer 1 ──> Layer 2 ──> Layer 3 ──> Output
│ ▲
Non-Sequential (Game/Graph): Input ──> Layer 1 ─────────────────┘ ──> Output
└───> Layer 2 ─────────┘
Multi-Player Dynamic Games
Traditional control-theoretic frameworks view neural network training as a sequential trajectory optimization problem. To handle the non-Markovian dependencies introduced by skip connections and parallel branches, Liu et al. introduced DGNOpt (Dynamic Game Theoretic Neural Optimizer). Instead of treating training as a single sequential pass, they reframe the network as a multi-player dynamic game where each layer is an individual player. Optimization then becomes a matter of finding a Nash equilibrium among all layers simultaneously, naturally accounting for non-linear, non-adjacent paths.
Statistical Mechanics and Memory
Another approach borrows from physics. Venturi et al. apply the Mori–Zwanzig formalism—a method from statistical mechanics used to reduce complex dynamical systems—to deep learning. By treating the non-sequential routing of information as a system with "memory," this framework provides a formal mathematical path to decompose complex, non-linear information paths into manageable components, offering a way to simplify deep architectures without losing critical information.
Non-Monotonic Information Metrics
If standard mutual information behaves unpredictably due to the breakdown of the DPI, how can we reliably track a network's internal learning dynamics?
One solution is to use alternative metrics like Sliced Mutual Information (SMI). As Wongso et al. demonstrate, SMI can capture the "growth" of usable information across successive deterministic layers without being strictly bound by traditional DPI constraints. This allows researchers to characterize layers not merely as passive, lossy filters, but as active geometric concentrators of task-relevant information.
Similarly, tracking the flow of Fisher Information—as explored by Weimar et al.—allows for the monitoring of parameter-relevant information preservation even when data bypasses layers entirely, providing a model-free way to detect overfitting without a validation set.
Conclusion
The shift from sequential, Markovian models to non-monotonic graph topologies represents a major evolution in how we conceptualize deep neural networks. As highlighted by Muratore et al., modern non-sequential architectures challenge our very definitions of receptive fields and linear hierarchies. By moving away from the assumption of strict informational decay and embracing tools from game theory, statistical mechanics, and advanced information theory, the machine learning community is building a more accurate, robust foundation for understanding how modern deep networks actually process information.
Key Foundational Work
On the Information of Feature Maps and Pruning of Deep Neural Networks (2020) — contributed 3 concept elements to this direction.
Information Flow Through U-Nets (2021) — contributed 2 concept elements to this direction.
Dynamic Game Theoretic Neural Optimizer (2021) — contributed 2 concept elements to this direction.