TL;DR
This paper introduces a novel neural network architecture that dynamically allocates computational resources based on input difficulty, inspired by human cognitive "System 1" and "System 2" thinking. By using an autoregressive model to predict input "surprisal" (unexpectedness), the system intelligently switches between a small, fast network for easy inputs and a larger, slower network for challenging ones, achieving comparable performance to a always-on large network with 15% fewer computational operations (FLOPs).
What Problem Does This Paper Address?
Modern deep neural networks, despite their power, often expend the same amount of computational effort on every input, regardless of its complexity. This is akin to using a powerful calculator for simple arithmetic – it works, but it's inefficient. The authors draw an analogy to human cognition, where "System 1" (fast, automatic) handles routine tasks, and "System 2" (slow, deliberate) is engaged for difficult or surprising situations. They aim to bridge this gap in artificial intelligence by developing neural networks that can similarly adapt their computational load to input difficulty, thereby improving efficiency without sacrificing accuracy. This is particularly relevant as neural networks grow larger and more computationally expensive.
Methods
The core of the proposed system is a surprisal-triggered conditional computation mechanism. It consists of three main components:
- Autoregressive Model: This component acts as a "System 1" predictor. It processes a sequence of input observations (e.g., audio frames) and learns to predict the next observation. This model is trained in an unsupervised manner on a large dataset.
- Surprisal Calculation: The "surprisal" of an input is measured as the negative log-likelihood of the actual observation given the autoregressive model's prediction. High surprisal indicates an unexpected input, suggesting higher difficulty.
- Controller and Dual Networks: A controller module takes the calculated surprisal as input and, using a sigmoid function, outputs a probability ($P_{big}$) of activating a "big, slow network" (analogous to "System 2"). Otherwise, a "small, fast network" (analogous to "System 1") is used. The controller can be trained to achieve a desired average computational budget (e.g., by targeting a specific mean and variance for $P_{big}$).
Crucially, the autoregressive model serves a dual purpose: it extracts features from the input and provides the surprisal signal. The authors chose to keep the autoregressive model's weights frozen during downstream task training to ensure it reliably computes surprisal.
Datasets
The experiments were conducted on two speech recognition datasets:
- TIMIT: A 3-hour dataset with hand-aligned phoneme labels.
- Mini-LibriSpeech: A 5-hour subset of the larger LibriSpeech dataset, which has transcripts but no phoneme labels. For this dataset, phoneme labels were generated using the Montreal Forced Aligner.
The autoregressive model itself was trained on the much larger, unlabeled LibriSpeech dataset (960 hours) to improve its feature extraction and prediction capabilities.
Key Insights and Ideas
- Surprisal as a Computational Budget Indicator: The paper demonstrates that input surprisal, learned in an unsupervised manner, is a surprisingly effective heuristic for determining computational effort. This aligns with human cognitive theories.
- Dual Role of Autoregressive Models: Using an autoregressive model for both feature extraction and surprisal estimation amortizes the computational cost, making the overall approach more efficient.
- Pareto Optimality: The surprisal-based conditional computation approach consistently yields Pareto-optimal results, meaning it achieves performance comparable to or better than baseline models while using significantly fewer FLOPs.
- Robustness to Hyperparameters: The surprisal-based controller's hyperparameters (mean and variance targets for $P_{big}$) are more interpretable and easier to set than those of a learned controller, which is a significant advantage for training very large models where hyperparameter tuning is costly.
Key Results
- The proposed surprisal-triggered conditional computation model achieved performance comparable to a baseline model that always used the "big network," but with 15% fewer FLOPs on average.
- The model demonstrated lower variance in Phoneme Error Rate (PER) compared to a learned controller baseline.
- Ablation studies confirmed that using autoregressive features significantly improved PER over using raw input features.
- The approach showed robustness across different hyperparameter settings and neural network architectures.
Limitations and Future Work
- Uniform Reward Assumption: The current model assumes a uniform potential benefit from using the big network, regardless of the input difficulty. Future work could incorporate a more data-driven evaluation of costs and benefits, considering the expected reward of expending more cognitive effort.
- Single-Task Focus: The experiments were conducted on single-task scenarios. The authors suggest exploring multi-task settings where an agent must allocate a limited computational budget across various tasks simultaneously.
- Deterministic Execution: While the primary model uses stochastic sampling for selecting networks, the paper explores deterministic execution for testing. This revealed a slight increase in PER when training stochastically but testing deterministically, suggesting that stochastic training is crucial for avoiding overfitting.
- Hyperparameter Tuning: While the surprisal controller is less sensitive than learned controllers, further exploration of hyperparameter tuning for the autoregressive model itself is suggested.
Summary updated