Standard neural networks are static: they apply the same number of layers to every input, regardless of whether that input is a clear, iconic image of a cat or a blurry, occluded scene. This "one-size-fits-all" approach is inherently inefficient. Research into input-adaptive depth seeks to transform the network from a rigid pipeline into a flexible computational resource that scales its effort to the difficulty of the task at hand.
The Efficiency of Variable Effort
The core intuition behind this research direction is that not all data points require the same amount of transformation to reach a correct prediction. As Xiaonan Li and colleagues demonstrate in Accelerating BERT Inference for Sequence Labeling via Early-Exit, simple instances often reach a stable, correct classification in the shallowest layers of a model. Forcing these "easy" samples through the remaining 80% of the network is a waste of energy and time.
By allowing a model to vary its active depth, researchers have found a "triple win": improved efficiency, maintained accuracy, and sometimes even enhanced robustness. RDI-Nets exemplify this by routing ambiguous or adversarial inputs to deeper, higher-capacity branches while processing clean data through efficient, shallow paths.
Mechanisms for Dynamic Depth
How does a network decide when it has "done enough" work? The literature identifies four primary mechanisms:
1. Early Exiting
The most straightforward approach is to attach auxiliary classifiers to intermediate layers. If a classifier's confidence exceeds a threshold, the model "exits" early. Ji Xin and Jimmy Lin show in Early Exiting BERT for Efficient Document Ranking that this can speed up document ranking by 2.5x. However, traditional early exiting is rigid—it must process layers 1 through $N$ in order.
2. Dynamic Layer Skipping
To overcome the rigidity of early exiting, models like SkipNet, developed by Fisher Yu, use lightweight gating modules to decide whether to execute or bypass specific residual blocks. This allows the network to "hop" through its architecture, selecting only the transformations relevant to the specific input features. Similarly, SGAD uses a "soft guideline" network to teach a mask-generator which layers are redundant, reducing FLOPs by up to 77%.
3. Recursive Convergence
Instead of a stack of different layers, some researchers use a single recursive layer that unfolds until the representation reaches an equilibrium. Alberto Rossi and Markus Hagenbuchner describe this in Embedding of FRPN in CNN architecture, where the effective depth is determined at runtime by the complexity of the feature map's convergence.
4. Direct Input-to-Layer Connections
A more radical shift involves changing the topology of the network itself. Vaggelis Dorovatas and Alexandros Potamianos introduce Auto-Compressing Networks (ACNs), which replace short residual connections with long feedforward links from every layer to the output. This induces "auto-compression," where the network naturally learns to push information into early layers. Similarly, ISLaB allows inputs to skip directly to any hidden layer, enabling the model to simplify into a linear structure when deep non-linear processing is unnecessary.
Why It Works: Representation Saturation
The success of these methods relies on the observation of representation saturation. In deep architectures, the internal representation of an input often stabilizes long before the final layer. Shengkun Tang and Yao Li leverage this in MuE to accelerate vision-language models, detecting when layer-wise similarity indicates that further processing will yield marginal gains.
This principle extends beyond classification. In generative tasks, AdaDiff applies more layers to complex structural denoising steps and fewer to simple refinement steps. In sequential data, DA-LSTM uses a "portion gate" to skip calculations during low-information periods, focusing resources only on high-value transitions.
The Pareto Frontier of Adaptive Depth
A recurring theme across these papers is that a single adaptive model often outperforms a collection of static models. Woochul Kang and Hyungseop Lee show in Adaptive Depth Networks with Skippable Sub-Paths that adaptive architectures produce a more efficient Pareto frontier than individually trained static models of varying sizes.
By treating depth as a dynamic variable rather than a fixed hyperparameter, these researchers are moving toward "on-demand" neural networks: models that can be instantly reconfigured to fit the computational constraints of a mobile device or the complexity of a specific user query without needing to be retrained.