RAPTOR: Ridge-Adaptive Logistic Probes

Ziqi Gao, Yaotian Zhu, Qingcheng Zeng, Xu Zhao, Ziqing Wang, Feng Ruan, Kaize DingarXiv 2026 · arXiv

TL;DR

RAPTOR demonstrates that a simple, well-tuned $\ell_2$-regularized (ridge) logistic regression probe can match or outperform complex state-of-the-art methods for LLM activation steering. By treating the regularization strength $\lambda$ as a critical hyperparameter rather than a static stabilizer, RAPTOR extracts highly accurate and directionally stable concept vectors at a fraction of the computational cost of alternative methods.

What Problem Does This Paper Address?

In the field of LLM interpretability and control, "probe-then-steer" pipelines are increasingly popular. Researchers train a lightweight classifier (a probe) on the internal layer representations of an LLM to detect a specific concept (e.g., "sarcasm" or "truthfulness"). The weights of this probe define a "concept vector." During inference, this vector can be additively injected back into the LLM's activations to steer its behavior without retraining the entire model.

However, an effective probe must satisfy three competing demands:

  1. Accuracy: It must reliably predict the concept.
  2. Directional Stability: The learned vector direction must be robust to minor training data perturbations so it remains reusable.
  3. Computational Efficiency: Training must be cheap, as sweeps often span dozens of layers, models, and concepts.

Standard unregularized logistic regression fails here. In the few-shot, high-dimensional regime typical of LLM probing, activations are often perfectly linearly separable. This causes standard optimization to blow up the weight norms, leading to numerical instability and brittle concept vectors. While complex estimators have been proposed to fix this, they are often computationally expensive and slow.

Methods

Figure 1: Overview of the RAPTOR pipeline for additive activation steering: extract layerwise last-token embeddings, standardize features, fit an ℓ2-regularized logistic probe with λ selected on a validation split, rescale to embedding space and normalize to obtain the concept vector vℓ, then steer at inference by hℓ,T ←hℓ,T + αvℓ.

The authors introduce RAPTOR (Ridge-Adaptive Logistic Probe), a minimalist pipeline consisting of the following steps:

  1. Extraction & Standardization: Extract layerwise last-token embeddings and standardize features using train-only statistics to prevent information leakage and control feature scales.
  2. Ridge Logistic Regression: Fit an $\ell_2$-regularized logistic regression probe with an intercept.
  3. Validation Tuning: Crucially, the ridge regularization strength ($\lambda$) is dynamically selected via a validation split rather than being left as a fixed default.
  4. Folding Back & Normalization: Fold the parameters back into the native, unstandardized embedding space and normalize the weight vector to yield the final concept vector $v_\ell$.

To ground this empirically, the authors provide a rigorous high-dimensional theoretical characterization using the Convex Gaussian Min-max Theorem (CGMT). This mathematical framework analyzes the probe under an idealized Gaussian teacher-student model, proving how $\lambda$ acts as a direct "stability knob" that controls the signal-to-orthogonal-noise ratio of the extracted vector.

Datasets

The method was evaluated across a diverse suite of six human-written binary concept datasets:

  • STSA (Sentiment)
  • Cities (Factual knowledge)
  • Common (Common sense)
  • CounterFact (Counterfactual factual updates)
  • HateXplain (Hate speech detection)
  • Sarcasm (Sarcasm detection)

These datasets were tested across 7 instruction-tuned LLMs from the Llama, Qwen, and Gemma families at various scales.

Key Insights and Ideas

  • The Separability Trap: Probing datasets are frequently linearly separable in high dimensions. Without proper regularization, optimization algorithms continue pushing weights to infinity, making the baseline probe highly unstable in practice.
  • $\lambda$ is the Key Control Knob: Instead of viewing $\ell_2$ regularization as a minor numerical stabilizer, the paper reveals it is the primary driver of concept vector quality. Tuning $\lambda$ directly optimizes the trade-off between out-of-sample accuracy and directional robustness.
  • Simplicity Wins: Elaborate, specialized estimators are often unnecessary. A classical statistical method (logistic regression), when properly regularized and tuned, serves as a formidable and highly efficient baseline.

Key Results

Probe Accuracy

RAPTOR consistently matched or outperformed more complex baselines like xRFM and Gaussian Concept Subspace (GCS). Averaged over the entire model-dataset evaluation grid, RAPTOR achieved a 0.874 best-layer accuracy, outperforming GCS (+1.96 points) and xRFM (+0.29 points). The gains were most pronounced on harder semantic tasks like HateXplain and Sarcasm.

Figure 2: Accuracy differences across the full model–dataset grid (RAPTOR minus baseline) Left: GCS; right: xRFM. Top: avg accuracy (mean over layers); bottom: best accuracy (best layer).

Directional Robustness

Under a strict stability test (randomly dropping 20% of the training data over 20 ablation runs), RAPTOR demonstrated significantly higher absolute-cosine agreement compared to xRFM, remaining highly competitive with GCS while maintaining better accuracy.

Computational Cost

Because RAPTOR relies on standard logistic regression, its training time is orders of magnitude lower than complex alternatives. Wall-clock time measurements show RAPTOR is consistently the fastest method across all model scales, making massive layer-wise sweeps highly practical.

Limitations and Future Work

The primary limitation lies in the theoretical analysis: the mechanistic characterization using CGMT assumes an idealized Gaussian teacher-student model. While the structural predictions qualitatively align very well with real LLM embeddings, actual LLM activations deviate from pure Gaussian distributions and exhibit complex feature dependencies. The authors note that future work should extend this theoretical framework to encompass more realistic, non-Gaussian feature distributions.

Summary updated

Abstract

Probing studies what information is encoded in a frozen LLM's layer representations by training a lightweight predictor on top of them. Beyond analysis, probes are often used operationally in probe-then-steer pipelines: a learned concept vector is extracted from a probe and injected via additive activation steering by adding it to a layer representation during the forward pass. The effectiveness of this pipeline hinges on estimating concept vectors that are accurate, directionally stable under ablation, and inexpensive to obtain. Motivated by these desiderata, we propose RAPTOR (Ridge-Adaptive Logistic Probe), a simple L2-regularized logistic probe whose validation-tuned ridge strength yields concept vectors from normalized weights. Across extensive experiments on instruction-tuned LLMs and human-written concept datasets, RAPTOR matches or exceeds strong baselines in accuracy while achieving competitive directional stability and substantially lower training cost; these quantitative results are supported by qualitative downstream steering demonstrations. Finally, using the Convex Gaussian Min-max Theorem (CGMT), we provide a mechanistic characterization of ridge logistic regression in an idealized Gaussian teacher-student model in the high-dimensional few-shot regime, explaining how penalty strength mediates probe accuracy and concept-vector stability and yielding structural predictions that qualitatively align with trends observed on real LLM embeddings.
BibTeX
@misc{gao2026raptor,
  author = {Ziqi Gao and Yaotian Zhu and Qingcheng Zeng and Xu Zhao and Ziqing Wang and Feng Ruan and Kaize Ding},
  title = {{RAPTOR: Ridge-Adaptive Logistic Probes}},
  year = {2026},
  eprint = {2602.00158},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.00158},
}
Figure