Bidirectional Feature Representation in Latent Spaces

29 concepts11 papers

When we try to understand how a neural network represents the world, we often look for discrete, human-understandable "concepts." To find these concepts, mechanistic interpretability tools like Sparse Autoencoders (SAEs) and matrix factorization techniques typically enforce a strict non-negativity constraint. The intuition seems straightforward: a concept is either present (positive activation) or absent (zero).

However, this constraint introduces a fundamental bottleneck. The real world—and the internal state of neural networks—is full of continuous spectra, logical dualities, and polar opposites (e.g., hot vs. cold, male vs. female, or activation vs. suppression). When forced into a non-negative framework, a single continuous semantic axis is artificially fragmented.

As Xiaoqing Sun et al. demonstrate in Dense SAE Latents Are Features, Not Bugs, non-negative models are forced to allocate pairs of nearly opposite, or antipodal, features to represent a single continuous linear dimension. This artificial splitting inflates the required dictionary size, wastes model capacity, and complicates our understanding of how networks handle scalar values.

The alternative is bidirectional feature representation: allowing a single hidden dimension to take both positive and negative values to encode a full spectrum of meaning.

The Intuition of the Bipolar Axis

Think of a thermometer. Instead of building a device with two separate non-negative scales—one measuring "degrees of heat" and another measuring "degrees of cold"—we use a single continuous axis where zero represents a baseline, positive values indicate warmth, and negative values indicate cold.

In language and reasoning, representations naturally organize themselves this way. Yiben Yang et al. leverage this geometric property in Extracting Commonsense Properties from Embeddings with Limited Human Guidance by projecting word embeddings into spaces defined by opposing semantic poles (like big vs. small). This dual-pole approach defines a clear axis of comparison, vastly improving zero-shot relational reasoning.

Similarly, JinYeong Bak et al. show in Rotated Word Vector Representations and their Interpretability that when statistical rotation algorithms are applied to word vectors, the resulting dimensions naturally exhibit bipolar semantics. High positive values and high negative values cluster into distinct, contrasting conceptual categories rather than acting as simple binary presence-indicators.

This bidirectional behavior is not just a passive property of static embeddings; it is actively utilized by networks to execute rules. Miikka Silfverberg et al. discovered in Do RNN States Encode Abstract Phonological Alternations? that recurrent networks manage binary linguistic transformations through bipolar activation patterns in specific hidden dimensions. Positive and negative values along the exact same vector trigger opposing directions of a transformation, such as the strengthening or weakening of a phonetic feature.

Biological Precedent: Two-Tailed Encoding

While non-negativity constraints in artificial intelligence are often defended as being biologically inspired—mimicking the "all-or-nothing" spikes of biological neurons—neuroscience reveals that biology relies heavily on bidirectional coding.

Many biological neurons maintain a high baseline firing rate. This allows a single neuron to encode scalar values symmetrically: increases in activity represent excitation, while decreases represent active suppression. David Arkadir et al. documented this mechanism in the primate basal ganglia in Independent Coding of Movement Direction and Reward Prediction by Single Pallidal Neurons, showing that single units linearly sum and transmit complex value-based information through bidirectional shifts away from baseline firing.

This "two-tailed" strategy drastically expands a population's representational capacity. In the visual cortex, Rabia Gondur et al. demonstrate in A tale of two tails: Preferred and anti-preferred natural stimuli in visual cortex that V4 neurons do not just fire when their preferred visual feature appears; they are actively suppressed by specific, complex "anti-preferred" stimuli. By utilizing both excitation and active suppression, a biological neural network can achieve the same object recognition accuracy with a fraction of the units required by a traditional, one-tailed artificial network.

Unidirectional (One-Tailed) Encoding:
[Feature A: Male]   --> Activation > 0
[Feature B: Female] --> Activation > 0  (Requires two separate, redundant latents)

Bidirectional (Two-Tailed) Encoding:
                      Activation > 0 (Male)
[Unified Axis] ----+------------------------>
                      Activation < 0 (Female) (Single latent captures the full spectrum)

Unifying the Interpretability Toolkit

Recognizing that neural networks naturally organize information along bidirectional axes has led researchers to redesign unsupervised feature extraction architectures.

Even in early bioinformatics applications, capturing suppression proved vital. Liviu Badea and Doina Tilivea showed in Stable Biclustering of Gene Expression Data with Nonnegative Matrix Factorizations that by extending input matrices with inverted or "down-regulated" versions of features, non-negative factorization methods could successfully capture inhibitory or suppressive relationships within a unified framework.

In modern deep learning, the focus has shifted to modifying the activation functions of sparse autoencoders. In AbsTopK: Rethinking Sparse Autoencoders For Bidirectional Features, Xudong Zhu et al. introduce AbsTopK, an SAE variant that explicitly allows features to take both positive and negative activation values. By removing the non-negativity constraint, a single feature can represent bidirectional semantic axes. This prevents the fragmentation of logical dualities, improves reconstruction fidelity, and makes model steering significantly more effective while using fewer active features.

Symmetric, signed features also resolve deep theoretical issues in dictionary learning. Walter Nelson et al. introduce the iSAE in Toward Identifiable Sparse Autoencoders, proving that allowing features to have both positive and negative coefficients aligns better with classical signal recovery theory. This bidirectional approach prevents the model from needing separate, redundant atoms for opposing concepts, yielding more compact dictionaries that are statistically identifiable and stable across training runs.

To explore how these bidirectional codes scale, Romeo Valentin et al. developed DB-KSVD in DB-KSVD: Scalable Alternating Optimization for Disentangling High-Dimensional Embedding Spaces, scaling traditional dictionary learning algorithms to massive transformer representations to evaluate how bidirectional codes capture the underlying superposition mechanisms of large models.

Why Non-Negativity Fails for Steering Vectors

The practical danger of ignoring bidirectional features becomes obvious when attempting to interpret steering vectors—directions added directly to a model's latent states to manipulate its behavior.

As Harry Mayne et al. explain in Can sparse autoencoders be used to decompose and interpret steering vectors?, steering vectors frequently operate by "writing negatively" to certain feature directions to actively suppress specific concepts or behaviors. Because standard SAEs enforce non-negative reconstruction coefficients, they are fundamentally blind to these negative contributions.

This architectural limitation creates a severe masking effect: when a steering vector contains a negative projection onto a specific feature direction, a standard SAE cannot represent it. Instead, the optimization math forces the SAE to misrepresent this negative contribution as a positive projection onto an entirely different feature that happens to have negative cosine similarity with the original direction.

By forcing a bidirectional phenomenon into a unidirectional mold, traditional interpretability tools generate spurious features and masked interpretations. Transitioning to a signed, bidirectional framework resolves this confounding behavior, offering an explanation of model internals that is more compact, theoretically grounded, and faithful to how networks actually compute.

Key Foundational Work

Independent Coding of Movement Direction and Reward Prediction by Single Pallidal Neurons (2004) — contributed 2 concept elements to this direction.
Stable Biclustering of Gene Expression Data with Nonnegative Matrix Factorizations (2007) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 11 papers
  2. Version 1 · 2026-10-04 · 10 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_bidirectional_feature_representation_6a2a0f13,
  title = {bidirectional feature representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bidirectional-feature-representation/rdr_6a2a0f13ccf04c9295c8c00fe82639da},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_bidirectional_feature_representation_6a2a0f13_v2,
  title = {bidirectional feature representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bidirectional-feature-representation/rdr_6a2a0f13ccf04c9295c8c00fe82639da/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}

See something wrong? Report a mistake