Extracting Finite State Machines (FSMs) from Recurrent Neural Networks (RNNs) has long been a popular method for "interpreting" these complex black boxes. However, this paper warns that such extractions may yield "fool’s gold." By demonstrating that RNNs are highly sensitive to initial conditions and that the arbitrary choice of how we discretize state space can create illusory FSMs, the author argues that these extracted machines often reflect the observer’s bias more than the network's actual underlying logic.
The Problem with Discretizing Continuous Systems
In the early 1990s, researchers were highly optimistic about using RNNs to learn formal languages, such as those in the Chomsky hierarchy. Because RNNs operate on continuous internal states, their decision-making processes are notoriously difficult to interpret. To bridge this gap, a common practice emerged: "extracting" FSMs from trained networks by clustering the network's continuous internal state trajectories into discrete "buckets" or states.
John F. Kolen argues that this practice is fundamentally flawed. The core issue is a mismatch in system types: RNNs are continuous dynamical systems, whereas FSMs are discrete. Forcing a continuous system into a discrete framework risks obscuring the network's true behavior—specifically, how it handles chaotic dynamics, sensitivity to initial conditions, and varying levels of computational complexity.
Methods and Synthetic Models
Because this is a theoretical and foundational paper, it relies on mathematical proofs and simulations of dynamical systems rather than standard large-scale datasets. The author employs:
- Dynamical System Simulation: A custom-defined 3-unit recurrent network with a
tanhactivation function is used to demonstrate chaotic divergence. The author initializes a tiny "ball" of 1,000 points in the state space and tracks their evolution over time. - The Rotator Model: A mathematical abstraction of a point moving in a circular orbit, used to generate symbol sequences for complexity analysis.
- Critique of Clustering: An analysis of existing FSM extraction methods (like K-means or hierarchical clustering) explains why they fail to account for the exponential divergence of nearby states.
Key Insights: The Observers' Paradox
A central revelation of the paper is the Observers' Paradox. Kolen demonstrates that the computational complexity of a system (e.g., whether it generates a Context-Free or Context-Sensitive language) is not solely an intrinsic property of the system itself. Instead, it is heavily dependent on how the observer chooses to measure and partition it.
By simply altering the "decision boundaries" (the observation function) applied to a continuous system, the same physical dynamics can appear to jump between different classes of language complexity. This suggests that when a researcher "finds" a specific FSM in a network, they might merely be seeing their own partitioning strategy reflected back at them. Instead of viewing RNNs as "broken" versions of discrete automata, the paper argues we should view them as nonlinear iterated function systems.
Key Results: Sensitivity to Initial Conditions
In many RNNs, two states that start incredibly close together will eventually diverge exponentially due to chaotic dynamics. If you discretize the state space, this deterministic divergence manifests as a seemingly "nondeterministic" jump between FSM states, making the extracted FSM an unreliable map of the network's future behavior.
Visualizations of the network's state space powerfully illustrate this point. An initial cluster of points with a microscopic radius (e.g., $e < 0.01$) can completely smear across the entire state space within a small number of iterations. This rapid divergence proves that any discrete "state" identified by clustering is fleeting and inherently unstable.
Limitations and Future Directions
While the paper serves as a vital cautionary tale against blindly relying on FSM extraction, it is primarily diagnostic. It highlights what not to do but does not provide a definitive new tool for interpreting RNNs, other than advocating for a paradigm shift.
Furthermore, the examples utilize very small networks (e.g., 3 units). While the mathematical principles of chaos and divergence scale up, the paper does not explicitly explore how these issues manifest in the massive, high-dimensional networks used in modern deep learning—though it is highly likely that the "Observers' Paradox" becomes even more acute in hundreds or thousands of dimensions. Ultimately, Kolen suggests that to truly understand RNNs, the field must move away from discrete information processing abstractions and embrace the mathematics of continuous-space dynamical systems.
Summary updated