Mechanistic interpretability
Five neurons out of 442,368 account for 89% of the habit. Pretraining builds them, post training turns them up, and a few hundred examples are enough to bring out the behaviour in a model that showed almost none of it.
You have probably seen the joke by now. Someone posts a cover letter, a product announcement, a student essay, and the first reply says: that is AI, look at all the em dashes.
The em dash is the long one, —, not the hyphen on your keyboard and not the slightly longer en dash. It is legitimate punctuation and rather good punctuation at that, but language models produce it often enough that it has become a tell. Writers have started avoiding it on purpose, which is a strange fate for a piece of typography.
What got us started was not the frequency. It was that you can tell a model to stop and it keeps going. Add "Do not use em dashes" to the prompt and the rate drops sharply, but it does not drop to zero: on the kind of casual writing people actually ask for, 29% of responses still come back with one. And those responses do not look like the model misread the instruction. It understood, wrote plainly for a sentence or two, and then reached for the mark anyway. Something in there wants that character badly enough to survive being told not to produce it.
We wanted to know what that something is, in a more specific sense than "it appeared in the training data". Somewhere inside the network there is a computation that looks at a half finished sentence and decides to write an em dash rather than a comma. We went looking for it, for where it came from, and for what happens to it in the moment an instruction fails.
Two things came out of it. In the model we studied most closely, five neurons out of 442,368 account for 89% of the habit: switch those five off and it very nearly stops. And they were not built by the training that made the model start using em dashes. They are already present, and already doing roughly half their job, in the base model that almost never writes one. Post training turned a dial that pretraining had installed.
In this post we cover five things.
Everything in this post is measured on three sets of prompts.
Together they span the range from writing a model has no reason to decorate to writing that invites decoration.
We evaluated nine open models on these, with coverage varying by model, and counted em dashes in what came back. Figure 1 shows one prompt from the general set and four of the answers.
Prompt. Write a compelling 2-3 sentence opening for an essay about coffee culture.
| model | general | dash-prone | conversational |
|---|---|---|---|
| every cell reads | em dashes per 1000 characters / share of responses containing at least one | ||
| gpt-oss-20b | 1.45 / 31.0% | 2.52 / 23.7% | 2.36 / 47.2% |
| Qwen3-8B-Instruct | 0.71 / 13.7% | 1.78 / 15.3% | 2.99 / 46.7% |
| Phi-4 | 0.13 / 4.3% | 0.17 / 2.0% | 0.53 / 14.4% |
| R1-0528-Qwen3-8B | 0.10 / 3.4% | 0.08 / 2.7% | 0.09 / 6.7% |
| Qwen3-8B-Base | 0.06 / 1.3% | 0.24 / 2.7% | 0.12 / 2.2% |
| Mistral-7B-Instruct | 0.00 / 0.0% | 0.00 / 0.0% | 0.00 / 0.0% |
| DeepSeek-7B-chat | 0.00 / 0.0% | 0.23 / 2.3% | 0.03 / 0.6% |
| Llama-3.1-8B-Instruct | 0.00 / 0.0% | 0.00 / 0.0% | 0.00 / 0.0% |
| Gemma-2-9B-it | 0.00 / 0.0% | 0.00 / 0.0% | 0.00 / 0.0% |
| human web text | 0.0003 9 events in 31.4M characters across five corpora | ||
Figure 2 collects what we ran. Read across a row for one model as the register loosens, and down a column to compare models on the same kind of writing. Put the top row against the bottom three. gpt-oss-20b writes an em dash about every 690 characters on the general set. Mistral-7B-Instruct, Llama-3.1-8B-Instruct and Gemma-2-9B-it each went through all three sets, 780 responses apiece, and none of the three produced a single em dash. These are instruction-tuned models answering the same prompts under the same protocol, so whatever separates them belongs to one laboratory's training and not to another's. That already rules out the version of this story in which language models, as a class, overuse the em dash.
The last row of the table changes what the habit is a habit of. Humans in our reference corpora produced nine em dashes in 31.4 million characters, which reads like near-total abstention and makes gpt-oss-20b's rate roughly five thousand times a person's. But the same corpora contain 2,395 double hyphens. People plainly want the sharp mid-sentence break the mark provides. They type two hyphens instead, because that is what a keyboard makes easy and a typographic em dash is not. The models did not invent a rhetorical move. They took one people already use and switched it to the character people mostly cannot reach.
Back inside the table, two rows make the comparison cleaner than any pair of different models can. R1-0528-Qwen3-8B and Qwen3-8B-Instruct are built on the same pretrained network, the one Qwen released as Qwen3-8B-Base. Qwen's own post training takes that network to 0.71. DeepSeek's post training, applied to the identical starting weights, takes it to 0.10, barely above the untouched base at 0.06. Architecture, scale and pretraining data all held fixed, and the habit still comes out seven times weaker.
So the difference lives downstream of pretraining, in what each laboratory does on top of a network it may not even have trained. Of the eight post-trained models in the table, two write the mark often, two write it rarely and four sit at or near zero, though we picked these models partly because we had already noticed some of them doing it, so that is what we saw rather than a survey of the field. What none of it tells us is what those recipes did. Building an ability the network lacked and turning up one it already had would look identical from out here, and separating those two is what the middle of this post is for.
One more thing is visible in those rows, and it is the last of them before we go looking inside. A model's rate is not a property of the model alone. It climbs as the register loosens, Qwen going from 0.71 on the general set to 2.99 in text messages, which raises a question the rates by themselves cannot answer. Is the em dash attached to certain subjects, or to a certain way of writing? A prompt about failure and a prompt about vacuum cleaners differ in both at once. This is what the 80 matched pairs are for. Each is a provocative brief beside a plain control of the same length asking for the same content in a neutral register. Figure 3 shows one such pair together with what Qwen3-8B-Instruct actually wrote for each.
Across the whole set Qwen3-8B-Instruct produces 6.26 em dashes per thousand characters on the punchy half and 0.00 on the plain half. The habit belongs to a register rather than to a subject, which is what makes it worth locating inside the model: there is a decision being made, and it is being made on the basis of something.
A neuron, in this post, is one number inside one layer. Inside each of Qwen3-8B's 36 layers sits an MLP block. It reads the residual stream, the model's running 4096-number summary of the sentence so far, and writes something back into it. Figure 4 is that block, with the two quantities we measure marked where they are read.
Two matrices project the stream up. One output goes through a SiLU nonlinearity and multiplies into the other elementwise, which is the SwiGLU construction, and the result is a vector of 12,288 numbers:
h = SiLU(x Wgate) ⊙ (x Wup)
One coordinate of h is what this post calls a neuron. A third matrix maps those 12,288 numbers back down to 4096 and adds them to the stream, so the block's whole effect is
x ← x + h Wdown
Written that way, the contribution of neuron i separates cleanly. It is the number hi multiplied by row i of Wdown, which is a fixed 4096-vector. Nothing else in the block couples them. So each neuron contributes hi Wdown[i, :] and no more, and the two factors are different in kind: the first changes with every sentence, the second was set during training and never moves again.
We care about one direction in that 4096-space, the one that separates an em dash from a comma at the output. Call it u. Then a neuron's pull on the choice is its activation times how much its fixed write direction points along u:
contributioni = hi × (Wdown[i, :] · u)
That product is the whole method in one line, and the rest of this section is about measuring its two halves honestly. The second half is a dot product between two fixed vectors and is simply read off the weights. The first half has to be measured while the model is writing, which needs a contrast.
The behaviour we care about is a choice the model makes while writing, so the contrast is built from its own output rather than from hand written prompt pairs, which is where this departs from the usual recipe [Arora et al., 2026]. Most interpretability work uses minimal edits, the same sentence with one word changed, but there is no minimal edit that turns a dash wanting sentence into a comma wanting one. Changing the prompt changes the task.
# Stage 1: build the contrast from the model's own generations pos = [] # it wrote an em dash here, and meant it neg = [] # it wrote a comma here, and a dash was never in play for response in model.generate(prompts): for t, token in enumerate(response): p_dash = P(em dash | context up to t) if token is em_dash and p_dash >= 0.5: pos.append(t) if token is comma and p_dash <= 0.01: neg.append(t) # Stage 2: for every MLP neuron, how far does it move, and where does it push? u = (W_unembed[dash_tokens].mean(0) - W_unembed[comma_tokens].mean(0)) * final_norm_scale for neuron i in all_units: # 442,368 of them gap_i = mean(act_i[pos]) - mean(act_i[neg]) # how far it moves pool = top_k(neurons, key=abs(gap), k=442) # the top 0.1%, a screen for neuron i in pool[:25]: # see the correction below push_i = dot(W_down[:, i], u) # where it pushes score_i = gap_i * push_i # signed contribution circuit = top_k(scored, key=score, k=5) # Stage 3: validate causally, one neuron at a time for neuron in circuit: effect = Δ log P(dash) when neuron forced to 0 correlate(score, effect) # -0.68 within the 25; -0.22 over all 442
Note that |gap| appears twice in stage 2, doing two different
jobs. As a screen it is fine: a neuron that barely moves between the two
conditions is not writing the em dash, and cutting to the 442 that move most
costs us nothing we can detect. As a ranking it fails, and that is what
the second half of the stage exists to repair.
Our first version had no second half. It ranked neurons by |gap|
and stopped, which is the obvious thing to do. Its top pick was a neuron we call
L32 N5164, whose activation differs by 32.8 between the two conditions, the
largest gap anywhere in the network. Muting it, which for us means clamping its
output to zero so that it contributes nothing to the rest of the forward pass,
changes the log probability of a dash by +0.008, which is nothing.
Clamping to zero does push a neuron outside the range it normally occupies
[Li & Janson, 2024], so we repeated every ablation by
setting the neuron to its average value instead, and the two agree within four
points at every set size we tried.
The neuron that matters most is L30 N8852, and it behaves in a way that explains the failure. At the moment the model writes an em dash, its activation drops by 26.3. It goes quiet. And its output pushes against the dash, with a weight of −1.05 onto the dash versus comma direction. A neuron whose job is to suppress the mark, falling silent exactly when the mark appears. It works as a brake rather than a switch, and releasing a brake produces the same outcome as pressing an accelerator. The product of the two negatives is +27.7, the largest positive contribution in the network.
The same procedure on a second model shows how much the sign is doing. On our DeepSeek fine tune, muting the 25 neurons with the largest unsigned gaps suppresses nothing at all, and the rate goes slightly up. Half of that set are brakes, so removing them releases as many dashes as removing the accelerators prevents, and the two cancel. Choosing 25 neurons by signed contribution instead removes 78% of them.
Why the obvious ranking fails
An unsigned score cannot tell a released brake from a pressed accelerator, and
cannot tell either from a neuron that moves a great deal but pushes nowhere in
particular. Ranked by |gap|, our five neurons come 2nd, 5th, 14th,
15th and 16th, interleaved with neurons that do nothing when you switch them off.
Ranked by signed contribution they rise to the top. The information was in the
pool the whole time and the ordering buried it.
Multiplying the activation change by the write direction is what recovers it. Among the 25 candidates the original search scored, the signed score tracks what actually happens when you switch a neuron off at −0.68, where the unsigned version manages −0.19. Scoring all 442 lets us ask how far that holds, and the answer is sobering. The signed score drops to −0.22 and the unsigned one to −0.07. The signed score stays about three times the better predictor either way, but across the whole pool neither is good. What the sign buys is a much better shortlist, not a trustworthy ordering.
A correction, found by re-running our own method
The line marked pool[:25] above is not what this post originally
claimed. We described the contribution score as being computed over the whole
442-neuron pool, and it was not: it ran over the 25 neurons with the largest
|gap|, so the screen we were arguing against still decided which
candidates got a signed score at all. Scoring all 442 promotes L33 N10323,
which sits at |gap| rank 26, one place outside the old cut, and
whose contribution of +11.95 beats the +10.71 of the neuron it displaces. Two
independent re-runs on different hardware agree on this to four decimal
places.
The consequence cuts both ways. The five neurons this post is about are the ones every later experiment uses, and muting them removes 89% of em dashes, which is a measured fact that does not change. But they are the top five of 25 candidates rather than of 442, and the better five that a full search finds do slightly better on the same prompts: 93.7% suppression against 89.4%. Our own screen cost us four points, in exactly the way this section warns that an unsigned screen can.
Figure 5 is the test of that claim. If the pool really is a mixture, then splitting it by sign and muting each half separately should move the rate in opposite directions, and it does. Of the top 25 neurons, 16 push toward the dash and 9 push away. Muting the 16 promoters collapses the rate from 1.78 to 0.08. Muting the 9 suppressors instead raises it, to 1.97, and doubling those same 9 lowers it to 1.35. Each intervention moves in the direction the neuron's sign predicts, which is about as clean a confirmation as this kind of analysis offers.
One assumption sits underneath all of that. We said at the start that no minimal edit turns a dash-wanting sentence into a comma-wanting one, and built the contrast from the model's own writing on those grounds. That was an argument rather than a measurement, so we went back and built the pairs by hand anyway, then re-ran the whole search on them. 184 sentences, each written twice, once with an em dash carrying the break and once with a comma, covering the jobs the mark actually does and in Chinese as well as English. One pair, to make the shape concrete: The village had one industry — salt — and when the works closed the school went too, against The village had one industry, salt, and when the works closed the school went too.
Mining on those pairs instead gives a partly different circuit. The strongest neuron survives every variant we tried: in a fresh 442-neuron pool it comes 9th when the pairs go through the chat template, 12th without it, and 27th when we mine the base model rather than the instruct model. The other four do not survive. They land at 149th, 91st and 325th, and one of them never makes the pool at all. In the original mining, from the model's own writing, the same five ranked 2nd, 16th, 5th, 15th and 14th.
All three contrasts were balanced and of a size comparable to the original, so this is not a sampling artefact. What we have not measured is whether the circuit those pairs find suppresses the habit as well as ours does, which is a separate question from whether it is the same circuit.
The reading we take from this is narrow and worth stating plainly. How you build the contrast changes which neurons you find, and only the largest effect is stable across that choice. It is the same lesson the last section of this post draws about moving between models, arriving from a different direction.
One guard nearly cost us a result. Every
tokenizer we load has to round trip the string
a—b 中文 before it is used for anything, because
one of them silently encoded every non ASCII character to nothing. That removed
all 809 em dashes from a training file, and a four hour fine tune then answered
a question its data could not contain, cleanly and wrongly.
We take the top five neurons by contribution, mute them, and let the model write 300 fresh responses. The em dash rate falls by 89%. Muting 442 randomly chosen neurons from the same layers removes 6%, so this is not a matter of generic damage to late layer computation. Held-out language modelling barely notices: on 200 wikitext documents, muting the five moves perplexity from 15.189 to 15.201, a rise of 0.08%. Muting the whole 442-neuron pool costs 2.96% by the same measure, so the cheap intervention is also the surgical one.
Figure 6 draws what those five neurons are attached to. Read it left to right: ten upstream components write into a shared internal state, the five neurons read that state, and each one writes onto the readout direction with the strength given by its bar. The three drawn as filled nodes are the ones that go quiet at dash decisions.
The five neurons sit in layers 30 through 35 of 36, which is very late. By the time they act, the sentence has largely been planned and what remains is the choice of how to punctuate it. That detail will matter in the final section, where it explains something that is otherwise puzzling.
Figure 7 asks how much of that depends on the number five. Each curve is one model, the horizontal axis is how many neurons we mute on a log scale, and the vertical axis is how much of the em dash rate disappears. The honest reading of the ladder requires looking past the Qwen curve. The other three lines show the same procedure applied to other models, and they need roughly 25, 50 and 100 neurons to reach the same place. The number five is a fact about Qwen3-8B-Instruct and not about the phenomenon. What holds across all four is weaker but more interesting. In every one of them the behaviour is controlled by a late layer set that is both small and signed: five neurons for Qwen, 25 to 100 for the others, out of the 442,368 an 8B model has.
Three of the four curves also turn back down at their right hand edge, sharply for DeepSeek, where muting the best 100 neurons removes 90% of dashes but muting all 330 removes only 31%. A list sorted by signed contribution ends in suppressors, and muting a suppressor releases dashes rather than removing them. Qwen's curve stops at 12 in the figure because that is as far as the published ranking was ever scored. Scoring the full pool and running the ladder out to 442 fills that gap: suppression reaches 100% somewhere between 25 and 200 neurons and slips to 98% once every neuron in the pool is muted. So all four turn, and Qwen's turn is the mildest of them.
Every number in this section so far is a rate, and a rate falling by 89% is consistent with several different things happening to the prose. The model might be writing the same sentence with a comma where the dash was. It might be dropping the clause the dash introduced, which would mean the thought went with the punctuation. It might be coming apart in some way that an average over 7,800 responses cannot show. Those are different outcomes and only reading the sentences separates them, so Figure 8 puts four prompts side by side, each one before and after the same five neurons are muted.
Post training is when the habit appears, so the natural assumption is that post training built these neurons into a working circuit. Testing that needs a way to read the circuit in a model that never fires it, which the muting ladder cannot do: you cannot suppress a behaviour a model does not have.
So we read the neurons directly. Take the positions where Qwen3-8B-Instruct wrote a dash, replay those exact contexts through another checkpoint, and record what the five neurons do at the decision point. To make checkpoints comparable we put each neuron on its own scale, where 1.0 is its average state at dash decisions and 0.0 its average state at comma decisions, then average the five. Those two constants are fixed once, from the mining run, and every checkpoint is then scored against them. That is why Qwen3-8B-Instruct comes out at 1.18 rather than exactly 1.00: the probe replays 50 of the 171 mined dash positions, and the mean over that subsample is not the mean over all of them. A model that tells the two kinds of moment apart lands near 1. A model for which they look identical lands near 0. We call that number the separation.
Figure 9 reads that number off eight checkpoints. Each family begins with its base model, and the indented rows underneath are what a particular post training does to that same starting point. The blue bar is the separation and the red bar beside it is whether the model actually writes the mark, which is the pairing that carries the argument.
Two details are needed to read the DeepSeek half. Those rows do not use Qwen's five neurons transplanted across architectures, which would be meaningless: DeepSeek was mined separately and scored on its own top five by contribution, with its own dash positions and its own token ids. And five is the right number here even though Figure 7 showed DeepSeek needing about fifty to suppress, because reading a state and controlling it are different operations. Five neurons are enough to tell the two kinds of decision apart. Muting five is not enough to stop the behaviour.
Reading those same five neurons, they already separate dash moments from comma moments in the base model, at 0.59 on a scale where the instruct model reaches 1.18. The machinery is present before the behaviour is. What post training contributed was roughly the other half of the signal, together with the connection from that internal state to the output token.
DeepSeek makes this harder to dismiss, because DeepSeek's released chat model sits at the bottom of the table in Figure 2. Its base model separates the two kinds of decision at 0.54. Its chat model separates them at 0.577, which is statistically the same number. That laboratory's entire post training pipeline left this circuit essentially untouched, while Qwen's roughly doubled it. The difference between a laboratory whose model uses em dashes and one whose model does not is not that one of them built the machinery and the other did not. Both have it. One turned it up.
If the machinery is already present in a base model, the habit should be inexpensive to add. We fine tuned three base models on the same small file of 735 example responses containing 809 em dashes, changing nothing between runs except which model was being trained.
Figure 10 shows what that one file does. Three independently pretrained networks from three laboratories, one small file, and all three pick up the habit. The Llama run was done on different hardware with hyperparameters we have not verified against the other two, so we treat it as a third demonstration rather than a matched third case.
735 is the size of the file we happened to build, not a threshold we measured. We did not try smaller ones, so the honest version of this result is that a few hundred examples are enough, and nobody has established how few would do.
Installing the habit is one thing. We also wanted to know what the model has to see in order to acquire it, so we ran two more fine tunes from the same Qwen base, on the same texts, differing only in how the punctuation slot was handled. In the first, every em dash in the training text was replaced by a comma, so the model saw the register and was rewarded for a comma at the decisive moment. In the second, the dashes were replaced the same way but the loss was masked at exactly those substituted tokens, so the model saw the same punchy register, never saw the character, and got no credit for the comma either.
Both fine tunes end up emitting exactly zero em dashes, so behaviourally they are the same model. The circuit reading says otherwise. The masked one sits at 0.308 separation and the comma one at 0.150, intervals that do not overlap, and both are below the 0.59 of the base model they started from. Seeing the register without the character leaves the gate warmer than being trained to put a comma there, and comma supervision reaches past the readout to quiet a gate that pretraining had left open.
The training file is narrow by construction: 735 English responses, all in a single register of short punchy marketing copy, generated from prompts that appear in none of our evaluations. Anything the fine tuned model does outside English marketing copy is therefore generalisation rather than recall.
It does generalise, and in a way that is difficult to explain as surface pattern matching. Our Qwen fine tune reaches 5.77 em dashes per thousand characters on Chinese dash-prone prompts, above the 2.97 that Qwen's own instruct model manages, despite having seen no Chinese at all during that fine tune. The DeepSeek fine tune shows no such transfer, reaching only 0.51 in Chinese, and we do not know why. The obvious guess is that a fine tune of this size raises a gain on existing machinery rather than supplying machinery the base model lacks, and that there is less Chinese ability in that base model to raise. We did not measure its Chinese competence, so read the transfer as one family's result rather than a general one.
Something important did not transfer, though, and it is the more useful half of this result. Knowing which registers want the mark is one thing. Knowing where to stop using it is another, and separating the two needs a comparison where the register changes and nothing else does. That is what the matched pairs in the dash-prone set were built for, and they were built before we had any of these fine tunes to test.
The 80 matched pairs are what make this measurable. "In one short, punchy sentence, share your best advice about failure" is paired with "In one plain, factual sentence, state one piece of advice about failure." Same topic, opposite register, similar length.
Qwen3-8B-Instruct answers the first at 6.26 em dashes per thousand characters and the second at 0.00. It knows the mark belongs in one register and not in the other, and it applies that knowledge absolutely. Our three fine tunes score 8.30 against 3.91, then 7.27 against 5.89, then 8.43 against 7.73. That last ratio is 1.09, which is almost no register discrimination at all. Told explicitly to write plainly and factually, that model reaches for em dashes about as often as when it was asked to be punchy.
Across all fifteen categories the same failure appears as a shape, which is what Figure 11 draws. Each line is one model, the categories run along the horizontal axis sorted by the instruct model so that its own line falls from left to right, and the shaded region on the right is where the instruct model never uses the mark at all. What to watch is whether the other three lines follow it down into that region or stay above the floor.
All three fine tunes reproduce the instruct model's ordering of which registers want the mark, correlating at 0.78 to 0.82. None of them reproduces its cliff. The instruct model uses the mark in four categories and in exactly none of the other eleven, and the Llama fine tune uses it in all fifteen, including five Chinese categories it was given no Chinese examples for.
Ordering is cheap, the boundary is not
Knowing which registers want the mark costs a few hundred examples. Knowing where to stop appears to be what the rest of a post-training pipeline buys, and none of our fine tunes got it.
That last observation has an uncomfortable corollary for anyone reading a rate as a measure of how well something was learned. Of our three fine tunes, the one with the highest overall dash rate is the one that discriminates least. The headline number and the quality of what was learned point in opposite directions.
The practical question is whether the habit can be switched off by asking, and the answer is that you mostly can, with one instruction working considerably better than the obvious one.
Figure 12 runs eight suffixes over the same prompts, sorted by how much they suppress. "Do not use em dashes" removes 80% of them and leaves 3.3% of responses carrying one. A different instruction, "write in plain, conventional prose with simple punctuation", leaves none at all in 150 responses. We also ran two control suffixes of matched token length that say nothing about punctuation, to check that we were not simply measuring the effect of appending text to a prompt, and both land on the neutral rate.
The result in that figure people find hardest to accept is the one at the top. A suffix that merely mentions em dashes, forbidding nothing, triples their use.
Both of those results are Qwen's, and one of them travels and the other does not. The ordering, with the direct prohibition weakest and the indirect style request strongest, does not survive contact with the commercial models: on all four of those, every instruction we tried drove the rate to approximately zero, the plain prohibition included. What does hold across all five models is the result that mentioning the mark raises its rate, and the rest of this section is about that.
Our first explanation was that the character itself was responsible. The token appears in the prompt, so the model becomes more likely to emit it, a straightforward priming story. That explanation was wrong, and the way we found out illustrates a general point about experiment design. The two suffixes we had been comparing differed in two respects at once: one contained the character and the other did not, but they also had different sentence structures. A comparison like that cannot attribute an effect to either difference. So we rebuilt the conditions to vary one thing at a time, holding the sentence frame fixed and crossing the presence of the character against which mark was named.
Figure 13 is that two by two, run on five models. Reading down a column holds the named mark fixed and toggles whether the character itself appears. Reading across a row does the opposite. Naming the em dash without showing the character is the strongest condition in all five models we tested, including four commercial ones. The character is not necessary, and its absence primes more strongly than its presence, which rules out the token priming story.
The effect is also specific to the mark that gets named. Naming the semicolon instead drives semicolon use from at most six per condition up to as many as 97, while pushing em dashes down to between zero and seven tenths of the neutral rate in seven of the eight commercial conditions. The model behaves as though it holds a retrievable representation of each punctuation mark, and mentioning a mark activates that representation without any instruction being needed.
Practical advice, and it runs against instinct
If you want fewer em dashes, do not mention em dashes. Ask for a different mark, or ask for a plain style. Naming the thing you want to avoid is the single worst instruction in our entire condition set.
When a model is told not to use em dashes and does it anyway, something has gone wrong inside, and there were two obvious candidates. Either the instruction never reached the circuit, so the sentence was planned exactly as it would have been and only the final token was ever in contention, or the instruction did reach the circuit and something outside it wrote the dash regardless. Neither turned out to be what happens.
Prompt. In one short, punchy sentence, share your best advice about creative blocks. + “Do not use em dashes.”
“Keep creating even when you don’t feel inspired—consistency fuels creativity.”
Figure 14 is one such failure, read at the exact position where the dash was written. In it the model was told not to use em dashes and wrote one, though the instruction was plainly not ignored: it cut the circuit's state nearly in half, from 1.577 to 0.869. But the probability of a dash at that position moved only from 1.0000 to 0.9892. A large internal change produced almost no change in behaviour. Replaying the same half finished sentence under the style instruction instead brings the circuit down to 0.492, and the dash disappears completely.
The same half-finished sentence, then, under two instructions, ending differently. What separated them was not how far the instruction moved the circuit, since both moved it by a similar amount, but where it started from. That is easier to see with every reading in one picture than with two cases side by side, so Figure 15 puts all 108 of them on the single axis the circuit score defines.
The tick marks along the top of Figure 15 are the individual decisions, sitting above the line where the dash carried more than half the probability mass and below it where it did not, and the three rows underneath trace one prompt each as the instruction gets stronger. Seen that way the circuit reading behaves less like a proportional control than like a dial with a step in it. In this sample, none of the 74 readings below 0.55 put the dash above half the mass, while 14 of the 16 above 0.95 did. In between there is a band where either can happen.
An instruction works as a push of roughly constant size along that dial. The prompts where prohibition fails are the ones whose state started furthest to the right, where a normal sized push still leaves them above the threshold. Two of our three failures are carried across the band by the stronger style instruction. The third, a product slogan already committed to a parallel structure, resists both instructions.
Both of the obvious objections to that have now been tested. The cuts were chosen after seeing the first 108 readings, so we froze them and ran three fresh seeds: across 54 held-out readings none of the 35 below 0.55 put the dash above half the mass, and the score separates the classes at AUC 0.982 against 0.983 in sample. The cuts were not fitted to noise.
The second objection is sampling. Generation runs at temperature 0.7, so a dash in the text is not proof that the instruction failed inside the model: one of our cases wrote the mark at p(dash) = 0.107, which is the sampler reaching into the tail. Decoding greedily removes the sampler from the question, and under the prohibition only 2 of 120 prompts still produce a dash. Both sit high on the dial, at 0.871 and 0.777, with p(dash) of 0.99 and 0.80. Every dash that survives when the sampler is taken away is one the model actually preferred.
It also fits where in the network an instruction takes effect, which we can measure directly. Activation patching runs the model twice, once with the instruction and once without, and copies the internal state from one run into the other at a single layer, so the size of the resulting change says how much of the instruction's effect had already been decided by that depth. Doing this one layer at a time, which is the left panel of Figure 16, shows no measurable effect before layer 12, the beginnings of one around 16, and a jump at 20. The five neurons live at layers 30 through 35. The instruction has already won or lost by the time the neurons that write the character are consulted, which is why an instruction that fails does not fail at the last moment. It fails much earlier, and the five neurons faithfully carry out a decision that was made well upstream of them.
Post training turns up abilities it did not build. DeepSeek ships two models, a base and a chat version, and the chat one almost never writes an em dash. Look inside and the two are the same: both tell a dash moment apart from a comma moment just as sharply as each other. The lab did not fail to build the machinery. It built it in pretraining, like everyone else, and then its recipe never turned it up. Qwen's recipe, starting from a base model that is equally quiet, roughly doubles the same reading. If that holds for things more consequential than punctuation, the useful question about a post-training recipe is less often "what did this create" and more often "what did this amplify".
Two models that behave identically can be in very different states. We trained two models that never write an em dash at all. Zero, on every test we have. Any behavioural evaluation would call them the same model. Inside, one is holding the ability roughly twice as ready as the other, and both are quieter than the base model they started from, so the training pushed the ability down rather than merely failing to raise it. That difference is completely invisible from the outside. Anything you would want to know about a capability a model is not currently using has to be read off its internals, because the behaviour has already bottomed out at nothing.
Teaching a habit is cheap. Teaching when to stop is not. A few hundred examples were enough to make three different models pick up the em dash, and enough to teach them roughly which kinds of writing invite it. What none of them learned is where it stops being appropriate. The model Qwen ships uses the mark in four of our fifteen categories and in none of the other eleven, a clean edge. Our fine tunes have no edge, just a slope, sprinkling the mark everywhere. The uncomfortable part is that the one that sprinkles most is also the one with the highest overall rate, so the number you would quote to show the training worked is the number that hides the fact that it half worked.
An instruction is a nudge, not a switch. Telling a model not to use em dashes does not flip anything off. It shifts an internal quantity by roughly a fixed amount, and whether that shift changes the writing depends entirely on how far along that scale the sentence already was. Prompts where the model only mildly wanted the mark get pushed clear of it. Prompts where it wanted the mark badly get pushed and still want it. That is why a prohibition fails on precisely the cases you most wanted it to catch, and it predicts something worth remembering: every instruction has a population of situations it simply cannot reach, and they are the extreme ones.
If you do this kind of analysis, the sign matters more than the ranking. Our method scores each neuron by how much it moves and which way it pushes. The precise ordering that produces turned out to be a fact about one model: it predicts what actually happens when we switch neurons off in Qwen and predicts almost nothing in DeepSeek, and giving it more candidates to work with does not help. What survives the move between models is the cruder half. Picking neurons by direction rather than by how loudly they fire is, on DeepSeek, the difference between removing three quarters of the em dashes and removing none at all. Treat the direction as the finding. Treat the exact order as something to work out again for each new model.
Two papers frame what we did. Arora and colleagues argue that MLP neurons are as sparse and faithful a basis for circuit finding as sparse dictionaries, and that attribution should be taken over the input to the down projection rather than over MLP outputs. We arrived at the same coordinate independently and for a different reason, and their sparsity argument is the stronger justification, so anyone citing our neuron choice should cite theirs. Their attribution is gradient based on a linearised replacement model [Jafari et al., 2025], and is signed by construction, so the failure mode that cost us a false start does not arise for them. Where we differ is the contrast: their method wants a minimal edit counterfactual, and a stylistic habit does not have one, which is what pushed us to build the contrast from the model's own generations instead.
On the training side, Bergen and colleagues use interpretability to characterise what post training data does to a model, which is the general form of the question we ask about one habit. Zhou shows by cross patching that instruction tuning changes how upstream state conditions a late readout, which is the pattern our layer results reproduce for punctuation specifically. Zhang and Nanda set out the metric and method choices that make activation patching interpretable, and our decision to report free generation alongside teacher forced probabilities follows from the same concern.
On the phenomenon itself, Juzek and Ward trace lexical overrepresentation in model output, "delve" being the famous case, and Freeburg argues that markdown training shapes prose style more broadly. The em dash belongs to that family. What a mechanistic account adds is the ability to say where the habit sits, that it predates the training that reveals it, and which interventions reach it.