Overview
Understanding what a neural network "sees" or "thinks" is a central challenge in AI interpretability. Typically, researchers use "inversion" techniques to find an input (like an image) that triggers a specific neuron. However, because of the Rectified Linear Unit (ReLU) function—which maps all negative values to zero—neural networks are "many-to-one" mappings. Many different inputs can produce the exact same output.
This paper provides a mathematical framework to explicitly calculate the preimage of a ReLU network: the complete set of all possible inputs that result in a specific set of internal activations. By treating the network's layers as geometric projections, the authors show that these preimages are not just single points, but complex "piecewise linear manifolds" that reveal exactly what information a network discards. Existing methods usually rely on heuristic optimization (gradient descent with priors) to find one plausible input. This paper addresses the gap by analytically determining the complete set, allowing us to see the full range of ambiguity and information loss inherent in the network's architecture.
Analytical Procedure and Geometry
The authors propose an analytical procedure based on linear algebra and geometry for fully connected layers.
- Hyperplane Definition: For a layer with $n$ nodes, each node's weight vector and bias define a hyperplane (a "null space") where the input to the ReLU is exactly zero.
- Basis Construction: They construct a specific set of basis vectors ($e_i$) for the input space. These vectors are derived from the inverse of the weight matrix $W$. Specifically, each basis vector $e_i$ is designed to be "seen" only by its corresponding neuron $i$ and ignored by all others.
- Preimage Characterization:
- If a neuron's output is positive, its input is uniquely determined.
- If a neuron's output is zero, its input could have been any value on the "negative side" of its hyperplane.
By combining these rules, the preimage of a single layer is defined as a linear submanifold. For multi-layer networks, these pieces stitch together to form a piecewise linear manifold in the original input space.
Key Insights and Information Loss
A key technical insight is that the basis vectors for the preimage are simply the columns of the inverse weight matrix ($W^{-1}$). This provides a direct link between the weights of the network and the geometry of its input-output mapping. Because ReLUs are piecewise linear, the entire preimage of a deep network is also piecewise linear. It can be thought of as a collection of flat "sheets" joined at the boundaries where different neurons toggle between on and off.
When a ReLU unit is "off" (outputting zero), it acts as a projection. The information about the input's magnitude in that specific direction is permanently collapsed onto a hyperplane. The preimage tells us exactly which "directions" in the input space the network has become blind to.
The paper successfully formulates a step-by-step algorithm to identify the indices of active/inactive neurons and generate the linear combination of basis vectors that define the preimage. The 2D visualizations qualitatively demonstrate how "irreversible mixing" occurs. Entire shaded regions of the input space can be mapped to a single point (like $(0,0)$), illustrating exactly how the network discards input variance to achieve abstraction.
Limitations and Future Work
While theoretically sound, the method has practical limitations:
- Scalability: The method requires inverting the weight matrix $W$. For modern deep networks with thousands of neurons per layer, this is computationally expensive.
- Layer Types: The current work focuses on fully connected layers. Extending this to convolutional layers (which have a specific sparse structure) or layers with pooling is a logical next step but remains future work.
- Empirical Depth: The authors note they are still in the process of empirically evaluating the procedure's ability to extract preimages in more complex, multi-layer scenarios beyond synthetic geometric examples.
Summary updated