Negative-Constraint Probes as Fingerprints for Identifying Black-Box Language Models

Suppose you can send prompts to a chatbot behind an API but cannot see its weights or token probabilities, and you want to know which underlying language model it is. Most attribution work classifies whatever text a model happens to produce. We propose instead to ask it things it is bad at, specifically instructions to avoid a concept ("describe a beach without mentioning sand", "name a material that is not a metal"). Models fail these instructions often, and the way each one fails is shaped by internal wiring that differs between models. A model may leak the forbidden word, swap in a near-synonym, or retreat to a distant substitute, and which of these it does depends on that wiring. We hypothesize that this pattern identifies a model more sharply, per query, than its answers to ordinary prompts.

The second idea is about representation. The failure is described at the level of concepts, not words. We place each response on a small graph of related concepts, with text embeddings attached to each node, so that the fingerprint records which part of the meaning space the model moved to around the forbidden concept, rather than which exact words it used. If this works, the fingerprint should survive paraphrasing. Paraphrasing is the attack that destroys current text-based detectors, because it changes surface vocabulary while keeping the meaning.

Core Claim

Under a fixed query budget and with no access to token probabilities, a model's answers to negative-constraint probes, encoded on a concept graph, identify it among many candidates more accurately than (a) its answers to matched affirmative probes and (b) a raw-text classifier trained on the same answers. This holds in particular when the answers are paraphrased and when the candidates share a base model.

The mechanism is borrowed from interpretability work. How Language Models Process Negation traced negation failures in Llama-3.1-8B and Mistral-7B to a conflict between two parts of the network. Middle layers build the correct complement (for "not gas", they activate "solid" or "liquid" in over 80% of cases). Late-layer attention heads then override this with raw word co-occurrence. Ablating those heads raised negative-prompt accuracy from 50.5% to 67.8%. That paper treats the shortcut heads only as a bug to remove. We treat the outcome of the conflict as evidence about the model's identity. Who wins, and which complement gets built, should depend on layer depth, the specific heads involved, and pretraining co-occurrence statistics, and those are exactly the things that differ between models.

Two related findings support the bet that the behavior is stable and model-specific. Controlled Text Generation via Language Model Arithmetic describes the "pink elephant" effect, in which naming a forbidden concept raises its probability. Cross-Task Generalization via Natural Language Crowdsourcing Instructions shows that negative demonstrations act as attractors. Both papers motivate the proposal rather than supply its method.

What Is New Relative to Prior Work

Two lines of work form the closest neighbors, and neither combines what we combine.

The first is passive attribution of text to its generator. TURINGBENCH and M4GT-Bench benchmark multi-class attribution, and MGTBench shows that paraphrasing drops a fine-tuned detector from F1 0.99 to 0.04. All three classify text the analyst did not choose. We choose the prompts, specifically to make the generators diverge.

The second is attribution through structural features. HLD uses lexical, syntactic, and semantic likelihood ratios and finds that syntax survives paraphrase. LexicoSynt learns syntax embeddings separately from word content for human authorship. Both look for invariance in grammar. We look for it in concept choice under a constraint, which is a different signal and one that can be elicited on purpose.

Probability-based forensics such as Every Language Model Has a Forgery-Resistant Signature and Reverse Engineering Configurations of Neural Text Generation Models give strong signatures, but they need full probability vectors or many samples. We use such methods only as an upper reference on open-weight models.

The graph representation reuses ideas from text-attributed graph learning, meaning graphs whose nodes carry text: GraphFormers and GLEM. We use them on a much smaller scale than those papers, which were built for million-node graphs. Our graphs have tens of nodes per response, so the scaling problems those papers address do not arise here.

Research Program

The probe set. We build the probe set in two tiers. The controlled tier extends the 648 "X that is not Y is Z" templates of How Language Models Process Negation to include morphological negation ("un-", "im-"), following the fragility documented in Assessing Step-by-Step Reasoning against Lexical Negation. The open tier contains generation tasks that forbid a word or concept, such as describing, explaining, or listing without it. For every negative probe we write an affirmative twin on the same topic with the same length requirement. This pairing is what lets us isolate the contribution of the negative constraint.

The response graph. For each response we extract content lemmas and link them to a fixed concept graph built from WordNet relations together with a nearest-neighbor graph over sentence-embedding space. The response graph contains the forbidden concept, the prompt's topic concept, the concepts the response mentions, and the shortest connecting paths among them. Each node carries a text embedding of its gloss. Each edge records its relation type, and the response-mentioned nodes also carry their position in the response.

A two-layer graph attention network encodes this graph into one vector per response. Vectors from the k probes sent to a system are averaged into a system fingerprint.

The graph is meant to capture three quantities: whether the forbidden concept leaked, how far the substitutes sit from it in graph distance, and which branch of the concept neighborhood the model chose. Because these are concept-level properties, a paraphrase that only swaps words should leave the graph largely unchanged.

Training. We train the encoder with supervised contrastive loss over fingerprints, following the metric-learning approach of CLULab-UofA at SemEval-2024 Task 8 and Team MGTD4ADL. This choice matters because a contrastive embedding lets us add a new model with a few enrollment queries and no retraining. That directly targets the failure on unseen generators reported in M4GT-Bench.

The causal test. On Llama-3.1-8B and Mistral-7B, we ablate the shortcut heads identified by How Language Models Process Negation and check whether the model's fingerprint moves, and by how much compared with ablating random heads. If the fingerprint is really produced by the conflict between middle and late layers, removing the late-layer heads should shift it substantially. A shift of that kind would also explain why the signal exists.

Baselines

The naive reference points are chance and asking a language model to judge. Chance is 1/N for N candidate systems. For the language-model judge, we give a strong instruction-tuned model the k responses and ask which of the N candidates wrote them. AIs Fail to Recognize Themselves and Mostly Think They are GPT or Claude reports 10.9% exact-model accuracy over 10 candidates for 100-word texts, with 94% of predictions going to GPT or Claude. M4GT-Bench reports PhD annotators at 21.2% on a four-way task. We expect this judge to be weak, and the method should beat it by a wide margin.

The strongest existing passive method is a fine-tuned RoBERTa classifier on raw response text. On TURINGBENCH, RoBERTa-Multinomial reports F1 0.81 over 20 classes. That number comes from a different generator pool and from passive news text, so we cite it only as context. We have found no reported result on Lacuna for active-probe attribution of modern instruction-tuned models.

We run RoBERTa on exactly our responses, with the same train/test split by probe template, the same k, and the same tuning budget (an equal number of hyperparameter trials on a shared validation split). We also run an HLD-style system, which builds lexical, part-of-speech, dependency, and SBERT features and passes them to XGBoost, following HLD. We include it because its paraphrase robustness is the bar to clear: it reports 97.04% AUROC under paraphrase, though for binary detection.

The key ablation runs every system, including ours, on affirmative-twin responses instead of negative-probe responses, with the same number of queries. We further ablate our encoder in three ways: we replace the graph with three scalar features (leak rate, mean substitute graph distance, substitute entropy), we remove the node text embeddings, and we remove the edges.

For open-weight models only, a probability-based upper reference uses the full-vocabulary signature of Every Language Model Has a Forgery-Resistant Signature.

Beating the baselines means two things. First, top-1 accuracy over the candidate pool must exceed the RoBERTa and HLD systems at k = 10 queries, with bootstrap confidence intervals that do not overlap. Second, the gap must hold or widen after paraphrasing, and it must be larger on negative probes than on affirmative twins.

Why not just…?

Why not just use ordinary affirmative prompts with the same query budget? This is the main threat. Models already have distinct styles, and M4GT-Bench shows near-perfect attribution of seen generators from ordinary text. If affirmative twins match negative probes at equal k, the negation part of the proposal contributes nothing, and the result reduces to probe-based attribution with a concept-graph encoder. The paired-twin design settles this directly. We measure accuracy as a function of k for both probe types, and our claim requires the negative-probe curve to reach a given accuracy with fewer queries. The claim also has to hold among fine-tuned siblings of the same base model, where stylistic differences are smallest. If the claim fails, we report that outcome.

Why not just fine-tune a text classifier on the responses? On clean responses a classifier may do well. But MGTBench shows such classifiers fall from F1 0.99 to 0.04 under paraphrase (MGTBench), and M4GT-Bench shows they fall below chance on unseen generators. Our paraphrase and enrollment experiments are designed to separate the classifier from our method on exactly these two failure modes. If RoBERTa survives paraphrased negative-probe responses as well as the graph encoder does, the graph is unnecessary.

Why not just count leaks and substitute distance with a few hand-made features? That may well capture most of the signal, which is why it is an explicit ablation. The graph is justified only if the choice of branch in the concept neighborhood (for example, "liquid" versus "solid" for "not gas") adds information that scalar features miss. The three-feature ablation measures that gap directly.

Why not just use probability signatures or ask the model who it is? Probability signatures need full-vocabulary log-probabilities, which commercial APIs withhold, so they serve only as an open-weight upper reference. Models also judge authorship poorly (AIs Fail to Recognize Themselves). Self-reported identity is trivially spoofed by a system prompt, so we treat it as a naive baseline and not as a competitor.

Proposed Experiments

Candidate pool. The pool contains roughly 15 to 20 systems. These include the generators named across the benchmarks (LLaMA-2, Llama-3.1-8B, Mistral-7B, BLOOMz, Dolly, StableLM, and API models from the GPT and Claude families), plus several fine-tuned variants of one Llama base. The variants are the hard case of shared corpora and shared architecture flagged by MGTBench. We sample each probe at two temperatures to test whether the fingerprint survives changes in decoding settings.

Splits. We split by probe template, so that test probes are never seen in training. For the unseen-generator test, we hold out three systems entirely and enroll each with k probes.

Metrics. We report top-1 accuracy and macro-F1 over systems as a function of k. We also report accuracy after an independent model paraphrases every response, and accuracy within the fine-tuned-sibling subset.

Decisive checks. The first is the negative-versus-affirmative gap at equal k, where a gap of zero rejects the core claim. The second is retained accuracy under paraphrase for the graph encoder compared with RoBERTa and HLD-style features. The third is the shortcut-head ablation: if ablating the heads identified by Zhou et al. moves Llama-3.1-8B's fingerprint more than ablating size-matched random heads, the fingerprint is tied to the mechanism we claim. Otherwise the signal is real but unexplained, and we will say so.