Operators and Curried Functions: Training and Analysis of Simple Recurrent Networks

Janet Wiles, Anthony BloeschNeurIPS 1991

This paper demonstrates that Simple Recurrent Networks (SRNs) can be "programmed" to perform complex logical tasks by using "hint units" during training. By viewing input patterns as operators and hidden states as curried functions, the authors bridge the gap between connectionist neural networks and formal functional programming.

The Black Box Problem

In the early 1990s, recurrent neural networks (RNNs) were gaining traction for their ability to process sequences. However, they were often treated as "black boxes." While researchers knew that weights stored the network's knowledge, it wasn't entirely clear how the internal dynamics represented complex, multi-step logic.

The authors address the challenge of explicitly programming internal representations. Instead of letting a network find any arbitrary solution, they wanted to see if they could force a network to adopt a specific computational structure—specifically, one that mirrors how a computer might evaluate a function with multiple arguments.

Methods and Datasets

The researchers utilized the Simple Recurrent Network (SRN) architecture. Their novel approach involved two main components:

  1. Hint Units: During training, the authors added extra target units to the output layer called "hints." These hints provided the network with intermediate information about the task (e.g., which boolean function was currently active). This "programmed" the hidden layer to preserve specific information necessary for future calculations.
  2. Canonical Discriminant Analysis (CDA): To understand what the network actually learned, the authors used CDA to project the high-dimensional hidden unit activations into a lower-dimensional space. This allowed them to visualize the "state transitions" of the network as it processed sequences.

The study used synthetic datasets consisting of Boolean sequences for the functions AND, OR, and XOR. Sequences followed the format {Function, arg1, arg2, ... argn}, where the first input selected the operation and subsequent inputs (0s and 1s) were the arguments. The network was trained on sequences with varying average lengths (0.5, 2, and 4 arguments) and tested on all possible sequences with 8 arguments, plus "stress test" sequences with 50 arguments to check for long-term stability.

Key Insights: Operators and Currying

The core theoretical contribution of the paper is a functional programming interpretation of neural network dynamics:

  • Inputs as Operators: The authors propose that inputs aren't just passive data; they are operators that transform the network's internal state. "Function" inputs (AND/OR/XOR) act as context-insensitive operators that "collapse" the hidden state into a specific functional region. "Argument" inputs (0/1) are context-sensitive; their effect on the state depends entirely on which function was previously selected.
  • Hidden States as Curried Functions: In functional programming, currying is the technique of translating a function that takes multiple arguments into a sequence of functions that each take a single argument. The authors argue that at any time step $t$, the SRN’s hidden state represents a "curried function" of the input sequence up to that point.
  • The Interpreter Metaphor: From this perspective, the weights of the network act as an interpreter. They don't just store a static mapping; they provide the machinery to execute the operations defined by the input stream.

Key Results

The network demonstrated remarkable success in learning and generalizing the underlying logic.

  • Perfect Learning and Generalization: In 3 out of 5 simulations, the network learned the training sets perfectly, with the remaining two achieving less than 1% error. Despite being trained on short sequences (average length 4), the network generalized perfectly to sequences of 50 arguments. This suggests the network learned the underlying logic rather than just memorizing sequences.
Finite State Machine learned by the network
  • State Visualization: The robust generalization indicated that the network had essentially learned a Finite State Machine (FSM). CDA revealed that the hidden unit space was organized into six distinct regions.
Canonical Discriminant Analysis of hidden states

These regions mapped perfectly to the six states required by a minimal FSM to solve the three-function task, proving that the "hint units" successfully programmed the desired internal representations.

Limitations and Future Work

While successful, the "programming" relied heavily on manual hinting—the researchers had to know exactly what intermediate information the network should represent. In more complex, real-world tasks, determining these hints may be difficult, and the paper does not explore how this method scales as the state space complexity increases.

However, the study provides a powerful framework for understanding RNNs. The authors suggest this perspective could be extended to natural language processing, where words act as operators on a mental state, offering a compelling bridge between connectionism and symbolic computation.

Summary updated

BibTeX
@inproceedings{wiles1991operators,
  author = {Janet Wiles and Anthony Bloesch},
  title = {{Operators and Curried Functions: Training and Analysis of Simple Recurrent Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1991},
}
Figure