Mechanistic Anomaly Detection via Internal Activation Trajectories

This direction is split into other directions: internal activation monitoring. Its last version stays readable and citable here.

Version 1 of 1204 papers284 conceptsfrozen 2026-10-04

Deep neural networks are notoriously overconfident. Even when presented with gibberish, adversarial noise, or inputs far outside their training data, the final softmax layer often produces a high-probability prediction. This "silent failure" occurs because the output layer is trained to map high-dimensional features into a narrow set of classes, often discarding the nuance of how it reached that conclusion.

Researchers have increasingly turned to Mechanistic Anomaly Detection (MAD) to solve this. The core intuition is that while an anomalous input might successfully "fool" the final output, it almost always leaves a distinctive, unnatural footprint in the network's internal activations. By monitoring the "trajectory" of data as it flows through hidden layers, we can detect failures that are invisible at the surface.

The Network as a Sensor Array

Instead of treating a model as a black box, this research direction treats every hidden layer as a sensor. In Data Sanity Check for Deep Learning Systems via Learnt Assertions, Haochuan Lu and colleagues argue that internal activation patterns are far more sensitive indicators of input validity than final outputs.

When a model processes familiar data, neurons fire in coordinated, predictable patterns—a "manifold" of normal behavior. Anomalies, such as adversarial perturbations, force the network into "low-probability" regions of this feature space. Jiajun Lu and David Forsyth demonstrated in SafetyNet that these signatures are particularly prominent in late-stage ReLU layers. By quantizing these activations, they created a "gradient-free" barrier that is difficult for attackers to bypass because the detector looks at the internal statistical distribution rather than the input pixels.

From Points to Trajectories

A single layer provides a snapshot, but the real power of mechanistic detection comes from looking at the sequence of transformations across the entire depth of the network.

Pierre Colombo and his team proposed a Functional Data Perspective where the layer-wise activations are treated as a continuous curve or "trajectory." They found that for in-distribution data, these transformations are "smooth." However, as Fran Jelenić explored in BLOOD, out-of-distribution (OOD) data often results in "jagged" or discontinuous transitions because the model lacks optimized pathways for those inputs.

This trajectory-based approach is unified in UniGuard by Anmin Fu. The researchers found that even if an attack (like a backdoor or adversarial example) mimics normal behavior in early layers to remain stealthy, it must eventually diverge in deeper layers to force the desired incorrect output. Tracking this divergence allows for near-perfect detection across diverse modalities.

Probing the Iterative Reasoning

In Large Language Models (LLMs), these trajectories represent the model's "iterative refinement" of a prediction. Nora Belrose and Zach Furman introduced the Tuned Lens, which uses linear probes to "read" the model's mind at every layer. They discovered that adversarial prompt injections induce a state of "internal conflict" or confusion that is clearly visible in the prediction trajectory, even if the final token generated seems plausible.

This concept extends to identifying "quirky" or deceptive behavior. In Mechanistic Anomaly Detection for "Quirky" Language Models, David Johnston and Nora Belrose showed that internal features can distinguish between a model using "honest" reasoning versus "cheating" heuristics. This suggests that MAD could be a vital tool for AI safety, allowing us to audit a model’s Deceptive Alignment by flagging when its internal reasoning process deviates from the norm.

Why It Works: The Invariant Gap

The fundamental reason these methods work is the "Invariant Gap." During training, a model learns specific invariants—correlations between features that always hold true for the target task.

  • Kang Choi and Jiwon Seo showed that when an input violates these learned invariants (e.g., a face with green hair), the internal channel correlations break down.
  • Fangzhen Zhao demonstrated in ADCP that by identifying "critical paths"—specific sequences of neurons that represent the functional logic of a decision—we can catch anomalies that appear normal at any single layer but follow an impossible sequence across the whole network.

By shifting the focus from what a model predicts to how the information propagates, Mechanistic Anomaly Detection provides a more robust, "white-box" layer of defense for deep learning systems.

Papers in this version

Showing 60 of 204 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_mechanistic_anomaly_detection_via_intern_ed4d9968,
  title = {Mechanistic Anomaly Detection via Internal Activation Trajectories},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mechanistic-anomaly-detection-via-internal-activation-trajectories/rdr_ed4d9968393b4024978778a6b2fb98a3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_mechanistic_anomaly_detection_via_intern_ed4d9968_v1,
  title = {Mechanistic Anomaly Detection via Internal Activation Trajectories},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mechanistic-anomaly-detection-via-internal-activation-trajectories/rdr_ed4d9968393b4024978778a6b2fb98a3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}