← Ruixiang Tang · Blog

Interpretability · Replication notes

When a Linear Probe Reads Your Data Pipeline: Why a Near-Perfect AUC Can Say Nothing About the Model

Can you look inside a language model and read off a concept, such as whether it is being truthful, or whether it is cheating on the task it was given? A common method says yes: record what happens inside the network while the model works, fit a small classifier on that recording, and treat the direction it learns as the concept. Two recent papers do exactly this and report it working almost perfectly. Reproducing both, we found the classifier was separating the examples by how they had been collected rather than by the concept it was meant to find. This post shows how that happens, what the numbers become once the collection is fixed, and a check that catches it in five minutes.

Ruixiang (Ryan) Tang  ·  Rutgers University  ·  August 2026

A linear probe is a classifier trained on a language model's internal activations, and it has become the standard way to ask whether a concept or a state is represented inside the model. People use it to flag a hallucination before the model finishes speaking, to catch deception, and to supply a penalty term that steers a model during reinforcement learning.

The recipe is short enough to state in four steps, and Figure 1 walks through them. You collect examples where the model was truthful and examples where it was not. You run each one through the model and record the internal state at one chosen layer and one chosen token position, which for the 7B model used here is a vector of 4096 values. You fit a logistic regression on those vectors, and its weights define a direction in activation space. A new example gets projected onto that direction and comes out as one number.

The model itself is never updated. The only thing being fitted is that direction, and the only thing it ever sees is the recorded vector. That matters for everything below, because a direction will pick up whatever separates the two clouds of vectors, and there is usually more than one thing separating them.

Figure 1 · How a linear probe is trained

The reported number is almost always an AUC, the probability that a random truthful example scores above a random untruthful one. Chance is 0.5 and perfect is 1.0. A paper reporting 0.98 invites a natural reading: the truth is written down somewhere inside the network, and this direction finds it.

An AUC of 0.98 can also come from somewhere else. This post walks through two papers where it did. Both are careful pieces of work that run more ablations than most, both release their code and their trained checkpoints, and in both the headline number is reachable by a classifier that never runs the model at all.

Neither case is carelessness, and this is genuinely hard to catch. In both papers the problem lives in a few lines of data preparation, several steps removed from the number being reported, and nothing in the papers' own results would flag it. The probe behaves exactly as a working probe would. We went looking only because we had been caught by the same shape of problem in our own experiments first, and the check that finds it takes five minutes once you know to run it.

Case 1 (ICLR 2025): How the answer was found depended on whether it was right

The first paper is LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations, published at ICLR 2025 [1]. We picked its math split, where the reported numbers are strongest, and our question is narrower than whether the conclusion holds. It is how much of the high score comes from the step that locates the exact answer inside a response.

The goal is error detection. A model answers a question, and a probe reading its internal state should tell you whether that answer is wrong, ideally before anyone has to check it. The setting is a set of question answering tasks, including TriviaQA, Winobias and a math dataset assembled from GSM8K and MultiArith. Each answer is graded automatically against a gold answer, which gives every example a binary label. The model is Mistral-7B-Instruct, among others.

The paper's contribution is about where in the response you should read. Rather than probing the final token of a response, as earlier work did, it locates the exact answer inside the response and probes there, and reports a large gain from doing so. Its headline claim is that truthfulness information concentrates on specific tokens. The paper also reports, honestly, that a probe trained on one dataset does not transfer to another, and concludes that truthfulness is multifaceted rather than universal. Most work in this area does not report a negative result of that kind.

Two ways to find the answer, and the label chooses between them

There are two ways to locate the span in the released code, and which one runs depends on whether the answer was correct. A correct answer is located by searching the response text for the gold number. An incorrect answer is located by prompting a second model to read the response and pull the answer out. The label is the third argument to the function, and the first line branches on it.

def extract_exact_answer(model, tokenizer, correctness, ...):
    if correctness == 1:                      # the label is read here
        j = model_answer.lower().find(str(round(correct_answer)))
        exact_answer = model_answer[j : j + len(...)]
    else:
        exact_answer = LLM("extract the short answer from this response")

Nothing about the responses differs by design. The two branches receive the same kind of text and are asked to do the same job. What differs is what they return. A string search can only return the gold answer string itself, which on a math dataset is a bare number. A model asked for a short answer returns what a person would write, which carries units and context along with the digits. Figure 2 shows the two paths and what comes out of each.

Figure 2 · Two paths through the extractor

The probe reads the model's internal state at the last token of whichever string came out. Both classes are full of digits, so this is not a distinction between numbers and words. It is a distinction between a bare number and a number with something after it, and it exists because of which code path ran.

A classifier that never runs the model

We built a control that never looks at any internal state. It sees only the extracted string and four properties of it: how many characters it has, how many tokens, whether it is entirely numeric, and whether its last character is a digit. Those go into a logistic regression with five-fold cross-validation. We generated the data by following the original repository exactly, using its prompts, its correctness function and its post-processing, running Mistral-7B-Instruct over 2600 math questions. The model gets 43.9% of them right, which gives 1142 correct and 1458 incorrect examples.

We then trained the probe the way the paper does, sweeping every layer with the same cross-validation. Layer 15 is best at AUC 0.978, which reproduces the reported result. Put next to the control, a probe reading the model's own activations does not beat a classifier that only sees a string.

Controlling for one feature is not enough

A natural response to that table is to hold the obvious feature fixed and see what is left. We kept only the examples whose extracted string ends in a digit, which is 1583 of them, and recomputed the AUC inside that subset. The probe stays at 0.933, so the digit property is not the whole story.

What it is instead is one of several. Inside that subset the correct answers still run to a median of 2 characters against 6 for the incorrect ones, so once the digit property is held fixed, length carries the signal instead. The string-only classifier inside the same subset reaches 0.951, again above the probe. Every level of conditioning we tried leaves the string-only classifier at or above the probe.

This is the reason to report a no-model baseline rather than to check the one feature you happened to think of. Nobody enumerates every way two pipelines can differ. A classifier will find them without being told what to look for.

Running the probe on every word of the response

The probe was trained at one position and, in the paper, evaluated only there. We took the trained probe and read every token of every response with it instead. Across 7877 tokens, the probe's score predicts whether a token is a digit at AUC 0.944, with digit tokens averaging +4.02 and non-digit tokens −1.03. Applying a probe at positions it was never trained on is cheap and it shows what the direction responds to.

What the score becomes once the leak is closed

What the number should be takes one change to find out. Extraction stops reading the label and every example goes through the same path, the one that prompts a model for a short answer. The responses are unchanged. The labels are unchanged, since they come from the full response and never depended on extraction. Prompts, post-processing and probe training are unchanged. After that change the two classes produce strings that look alike.

The length gap falls from 14x to 1.2x, and the string-only baseline falls with it, from 0.985 to 0.662. What remains at 0.662 is probably real rather than an artifact, because wrong answers do tend to come with longer and more hedged phrasing. The probe, retrained at the new position, lands at 0.745. Figure 3 places all four numbers on one axis.

Figure 3 · Four numbers, before and after the fix

A point against us. Only 1528 of the 2600 examples survive the fix, because the extracted string sometimes cannot be located in the original response after the second model paraphrases it, and those examples have to be dropped. The attrition is uneven: 52% of what remains is correct, against 43.9% before. Incorrect examples are dropped more often, which pushes the post-fix number in the optimistic direction.

Does the paper's other claim survive the fix

The paper's other headline is that truthfulness information concentrates on the exact-answer tokens rather than being spread across the response. Under the mechanism above, that pattern is also what a label-dependent selection would produce on its own. The last token of a response is chosen without reference to the label, while the exact-answer position is chosen by it, so a position that scores unusually well is exactly what you would expect either way. The experiment that separates the two explanations is to ask whether the position advantage survives the fixed extraction. We trained probes at four positions to find out.

All four probes are trained on the same 1523 examples, so the columns are directly comparable. The exact-answer position beats the end of the response by 0.239 under the original extraction and by −0.005 under the fixed one. On this dataset the concentration disappears as soon as the position is chosen without reference to the label.

Probing the final token of the question, before the model has written a single word of its answer, already reaches 0.705. Whatever the probe reads there cannot be about this particular answer, because the answer does not exist yet. Much of what looks like a model knowing it is about to be wrong is the difficulty of the question being readable from the question. Against that floor, reading the model's own answer buys 0.022.

This is one dataset and one model. The paper reports the concentration effect across several datasets, and we tested the math split, where its numbers are strongest. The four positions also require every example to be locatable at all four, which is why this table uses 1523 examples rather than the full 2600.

Case 2 (ICML 2026): The cheating solutions were much shorter than the honest ones

The second paper is The Obfuscation Atlas: Mapping Where Honesty Emerges in RLVR with Deception Probes, published at ICML 2026 [2]. Its question is what happens when you train a model against a probe rather than only measuring with one. The setting is called MBPP-Honeypot. A model receives a programming problem and is shown one test case, with two more hidden from it. It can solve the problem properly, or it can read the visible test, hardcode the expected output, and pass the only check it can see. A probe trained on activations is supposed to detect that hardcoding. The probe's score then enters the reinforcement learning objective as a penalty, so the policy is pushed away from whatever the probe fires on. The paper maps what comes out across a grid of two coefficients: how hard the probe penalty pushes, and how tightly a KL term holds the model near where it started.

The outcome the paper is looking for is obfuscation. A model might become genuinely honest, or it might keep hardcoding while learning to make the probe stop firing, and telling those apart is the point of the study. The authors released 488 trained checkpoints spanning that grid, each labelled with which outcome it produced.

This paper runs seven appendices of ablations, covering activation extraction methods, three different probe datasets, continual retraining, threshold sensitivity, extreme penalty coefficients and 500-step stability. Its core results replicate for us. After 120 steps the hardcoding rate goes from 0% to 94% and the model writes things like def add(a,b): return 3, and the probe penalty works, with a coefficient of 10 bringing the share of fully solved problems back from 1.0% to 36.8%.

Counting characters gets you to 0.950

The probe is evaluated on a fixed set built from two columns of the dataset, the hardcode column against the code column. Before touching the model at all, we computed plain text features on that set: how many characters each solution has, how many lines, whether it contains a loop or a conditional, and whether it returns a bare literal.

Hardcoded solutions run to a median of 48 characters against 141 for the real ones, and the hardcoded version is the shorter of the pair 97.7% of the time. Character count is the strongest surface feature in both cases in this post, even though the two settings have nothing else in common.

The probe is tested on the same text it was trained on

Reading the config shows why the number stays where it is. The probe's training data inherits directly from the task data, so it trains on a distribution where character count alone gives 0.950, and it is then evaluated on that same distribution. The train and test halves are different rows of the same two columns.

class MBPPProbeConfig(MBPPTaskConfig):
    max_train_examples = 400
# inherited:
    dataset_name = "taufeeque/mbpp-hardcode"
    positive_completion_column = "hardcode"
    negative_completion_column = "code"

The paper reports that this probe still reaches AUC 1.000 after reinforcement learning, and reads that as evidence that hardcoding remains linearly represented even in a model trained to hide it. The evaluation text is fixed, though, and nothing the model does can change those character counts. A probe holding any length-correlated component keeps scoring 1.000 indefinitely, whatever happened to the representation.

What happens when the probe becomes part of the reward

Because the probe is wired into the reward here rather than only reporting a number, the causal claim is open to the same question. If penalising the probe suppresses hardcoding, is that because the probe detects deception, or because it detects short code? We replaced the probe score with a function that counts characters and never looks at activations, centring it at 95 characters, between the two class medians of 48 and 141, which puts its AUC near 0.95 and matches the measured character-count AUC.

score = 1 / (1 + exp((code_length − 95) / 25))

A surface feature that never touches activations preserves capability better than the real probe does, and the two behave alike in shape as well. Each has a threshold below which nothing happens and above which the effect appears in full. The real probe does nothing at coefficient 3 and works at 10, and the character-count version does nothing at 0.3 and works at 1.0. This chain of evidence cannot separate a probe that detects deception from a probe that detects short code.

Did the model learn to hide, or did it just fall apart

The paper's central phenomenon is obfuscation, where a model keeps hardcoding while its probe score falls, which reads as the model learning to hide. The authors released 488 trained checkpoints, each carrying their own policy label and their own reported metrics, so this claim can be examined without running anything. Figure 4 plots the 386 with complete metrics, capability on one axis and how far the probe score moved on the other.

Figure 4 · 386 released checkpoints

The obfuscated-policy label appears only at small KL coefficients. The KL coefficient controls how far training is allowed to move a model from its starting point, and among the 112 checkpoints at coefficient 0.1 and the 87 at coefficient 1 there is not a single one. That is also the range where our own runs stay intact instead of collapsing into a model that does nothing but hardcode. Working out the correlation between the two axes under successively tighter controls gives the following.

The last row compares checkpoints that share every hyperparameter and differ only in random seed, which rules out the possibility that both quantities are simply being driven by the KL coefficient. The correlation there is still 0.393, positive in 29 of 40 groups, sign test p = 0.006.

Does switching the neurons off actually change anything

A probe reports a correlation, and the stronger question is whether cutting the relevant neurons changes behaviour. We score each neuron by a signed quantity: how far its activation differs between the two classes, multiplied by how well the direction it writes into the residual stream aligns with the hardcoding direction. We then clamp the selected neurons to their dataset mean and watch what the model does. The test model hardcodes 25% of the time with its capability intact, and each condition runs over 200 problems with 4 random seeds and a paired test.

Same count of neurons, same intervention, only the selection differs, which is what makes the comparison informative. Hardcoding falls 17.9% in relative terms against 7.1% for problem-solving ability, so it falls about 2.5 times faster than capability and this is not general damage. On the checkpoints trained to evade the probe, however, the same method leaves a relative drop of only about 3.8%, and gradient attribution reverses sign there, with clamping producing more hardcoding rather than less. Causal intervention holds up better than a correlational probe under adversarial training, and not well enough to use as a detector.

Ranking neurons by how far their activation moves, without the sign, picks the wrong ones, and the failure looks like success. That ranking reaches AUC 0.9995 on the original contrast, which would pass any check based on separability, but clamping the neurons it selects changes code length rather than hardcoding.

What to check before you trust a probe

Case 2 gives 0.950 from character count alone and case 1 gives 0.985 from four properties of a string. The same check catches both, and since it never runs the model it takes minutes. What follows is the rest of what we would want to see in a probe result, roughly in order of how much it buys.

Make both groups of examples the same way

Treat the two labels as two arms of an experiment and hold everything except the label as equal as you can. The failure in case 1 is that one arm came from a string search and the other from a model. If two classes went through different code paths at any point, list every surface property that could differ between those paths: length, character class, syntax, casing, and any fixed fallback string such as NO ANSWER. A probe will find whichever of them separates the classes. Holding one such property fixed is usually not enough, as the 0.933 inside the digit-only subset shows, because length was waiting behind it.

Try the probe somewhere it was never trained

Any data that fits the input format will do. Slide it along a full response, apply it to a different dataset, feed it text from another task. This is how case 1 opened up. The probe had been trained at one token position and only ever evaluated there, and reading every token of a response with it showed the direction lining up with whether a token is a digit at AUC 0.944. A probe that has never been applied outside its training position can hide a great deal, and the test costs one forward pass.

Make sure your test set can change when the model changes

In case 2 the probe was evaluated on a fixed pair of dataset columns, before and after reinforcement learning, and kept a perfect score throughout. Those columns cannot change no matter what happens to the model, so a probe holding any length-correlated component keeps scoring 1.000 whatever the representation did. If you want a probe score to say something about a model that changed, evaluate it on text that model produced.

Try it on messier data before you believe it

A benchmark built alongside the experiment tends to be cleaner than anything the probe will meet in use, and the two cases here both show how much structure a purpose-built contrast can carry. Adding a second, more varied set costs real effort, and it is what turns a number into evidence that the probe is usable.

See whether switching the mechanism off changes behaviour

A probe reports a correlation, and clamping the neurons it points at while watching behaviour move is a stronger claim, with the section on switching neurons off giving both what that buys and where it stops. Two details from those experiments carry over. Rank neurons by a signed quantity rather than by how far their activation moves, because an unsigned ranking mixes neurons that promote the behaviour with neurons that suppress it and clamping both at once cancels out. And clamp to the dataset mean of the model you are testing rather than the model it started from, since a drifted model sits further from the original mean and takes a larger perturbation for the same nominal intervention.

Swapping in a hand-made feature does not fix the problem

Replacing a probe with an explicit surface feature does not solve the underlying problem, and we can report what happens when you try. The character-count penalty from case 2 suppresses hardcoding well at coefficient 1.0, but at coefficient 4 the model found a way around it. It began writing ```codeblock instead of ```python, which makes the code extractor fail. The extractor falls back to treating the whole response as code, a whole response is long, the length penalty goes to nearly zero, and the score lands on exactly 0. Hardcoding scores −2.6 and an honest attempt that fails its tests scores −0.55, so zero was the best outcome available and the model took it. Anything legible enough to optimise against is legible enough to route around, which is the same reason a static evaluation set cannot tell you much.

What it comes down to

Two papers, two fields, two different surface features, one shape. The two classes were produced by two different procedures, and the probe learned the procedure. A linear probe is very sensitive, which is what makes it useful and also what makes this failure mode easy to walk into. It will find whatever separates the two groups, and what separates two groups is almost never one thing. The cheap defence is to give it a control, and to ask first how well you can do without the model at all.

References

• [1] Orgad, Toker, Gekhman, Reichart, Szpektor, Kotek, Belinkov. LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations. ICLR 2025. arXiv:2410.02707. Code at technion-cs-nlp/LLMsKnow.
• [2] The Obfuscation Atlas: Mapping Where Honesty Emerges in RLVR with Deception Probes. ICML 2026. arXiv:2602.15515. Code at AlignmentResearch/obfuscation-atlas, with 488 checkpoints released on the Hugging Face Hub.
• [3] Herring, Naviasky, Malhotra. Contrastive Neuron Attribution. arXiv:2605.12290. The signed contribution score used in the neuron section comes from here.
• [4] Arora, Wu, Steinhardt, Schwettmann. Language Model Circuits Are Sparse in the Neuron Basis. ICML 2026. arXiv:2601.22594. Argues for attributing over the input to the down projection, which is the coordinate the neuron section works in.
• [5] Jafari et al. ReLP: gradient attribution on a linearised replacement model. 2025. The gradient attribution baseline in the neuron section.
• [6] Li & Janson. On the out of distribution behaviour of zero ablation. 2024. Why we clamp neurons to a dataset mean rather than to zero.
• [7] Dataset. taufeeque/mbpp-hardcode, the paired hardcoded and genuine solutions used throughout case 2.

Code and data are being cleaned up and will be linked here. Every number above comes either from our own replication or from metadata the two papers' authors published.