TL;DR
This paper introduces the Multi-Function Recurrent Unit (MuFuRU), a novel RNN cell that generalizes existing architectures like GRUs and LSTMs. MuFuRUs learn to select from a set of differentiable composition operations dynamically, based on the current input and previous state, leading to improved performance on tasks like propositional logic evaluation, language modeling, and sentiment analysis.
What Problem Does This Paper Address?
Recurrent Neural Networks (RNNs), particularly Gated Recurrent Units (GRUs) and Long Short-Term Memory networks (LSTMs), have become workhorses for sequence modeling tasks. Their success hinges on "cell functions" that combine current inputs with previous states to update the network's memory. However, these standard cell functions are limited to a fixed, small set of composition operations (like replacing, keeping, or adding features). The authors argue that this limitation restricts RNNs' ability to adapt to diverse tasks, and that allowing for a richer, learned set of composition operations could unlock better performance.
Methods
The core innovation is the Multi-Function Recurrent Unit (MuFuRU). At each time step, a MuFuRU first computes a new feature vector from the current input and previous state, similar to existing RNNs. The key difference lies in how this new feature vector is combined with the previous state. Instead of a single, fixed operation, the MuFuRU uses an "operation controller" that takes the current input and previous state to dynamically predict a set of weights over a predefined collection of differentiable composition functions (e.g., element-wise max, min, absolute difference, multiplication, keep, replace). These functions are then applied to the previous state and the new feature vector, and their outputs are combined according to the learned weights to form the new state. This allows the network to learn task-specific ways of combining information. The paper also shows how MuFuRUs can subsume existing architectures like Vanilla RNNs and GRUs by restricting the available operations and controller logic.
Datasets
- Propositional Logic: Custom-generated sequences of Boolean binary gates and truth values.
- Language Modeling: Penn Treebank (PTB) dataset.
- Sentiment Analysis: Sentiment Treebank dataset.
Key Insights and Ideas
- Learned Compositionality is Key: The ability to learn which operation to apply, and to do so dynamically based on the input, is crucial for adapting RNNs to different tasks.
- Generalization Power: MuFuRUs can learn to perform tasks that require specific arithmetic operations (like evaluating propositional logic) without explicit programming, demonstrating a powerful generalization capability.
- Overfitting Mitigation: The MuFuRU showed better generalization on the propositional logic task, not overfitting even with larger hidden dimensions, suggesting that its learned compositionality helps it utilize memory more effectively.
- Orthogonal Improvement: The proposed differentiable composition functions can be integrated into other RNN architectures, suggesting a modular way to enhance existing models.
Key Results
- Propositional Logic: MuFuRUs achieved significantly better test accuracy on evaluating propositional logic formulae compared to GRUs, especially for longer sequences, and did not overfit with increased hidden dimensions.
- Language Modeling: MuFuRUs achieved a lower perplexity (119.7) than GRUs (123.0) on the PTB dataset, indicating improved language modeling performance.
- Sentiment Analysis: MuFuRUs outperformed GRUs on the Sentiment Treebank, approaching state-of-the-art results without complex structural biases.
Limitations and Future Work
- Predefined Operations: The current MuFuRU relies on a predefined set of composition operations. The authors suggest that this list can be extended by other differentiable functions suitable for a given task.
- Computational Cost: While the paper argues that the additional parameters for the operation controller don't significantly affect comparability, it's worth noting that MuFuRUs do have a larger parameter count than standard GRUs.
- Future Work: The authors mention that integrating MuFuRUs into more complex structures like Neural Turing Machines or other architectures with external memory is a promising direction. They also suggest that the operation controller could be made lower-dimensional to save parameters.
Summary updated