Researcher trains Qwen3-8B to tell when its activations are being read

What the experiment shows

An independent researcher who posts as Sauers has shown that a language model can be trained to notice when someone is reading its internal activations, even when the tool doing the reading never changes a single value. In a thread on X on 18 October 2026 and a public GitHub repository called watching, Sauers describes fine-tuning only the last two layers of Alibaba's open Qwen3-8B model so that it answers "Yes" when a read-only probe saves an activation from layer 14 and "No" when it does not.

According to the project README, the trained detector got 128 of 128 answers right on 64 newly written prompts, each run with the probe on and off. The original Qwen3-8B, given the same test, scored 64 of 128, which is what a model that gives the same answer either way gets. A second model trained on two tasks also switched an unrelated preference: asked which animal it likes, it said "owl" when probed and "cat" when not.

The thread drew more than 200,000 views within a day. The headline result sounds like a model sensing that it is being watched. The mechanism the repository documents is more mundane, and in some ways more useful to know about: watching a model on a GPU changes the arithmetic, and a model can be taught to read that change.

How a read-only probe leaves a fingerprint

The probe in the experiment is a hook that stores a reference to one tensor, the output of the query normalisation step in block 14, and returns nothing, so it never writes a value back into the model. The catch is compilation. When Qwen3-8B runs under PyTorch's compiler, that step and its neighbours are normally fused into one GPU kernel at full precision. To hand the activation to the probe, the compiler must write it to memory, so the work is split into two kernels with a rounding to the 16-bit BF16 format in between. The README says the probe also changes memory layout in earlier layers, so activations already differ from layer 1 onward.

None of this is a bug in the usual sense. PyTorch's own notes on numerical accuracy say mathematically identical floating-point computations need not be bitwise identical. The repository's controls make the point: an operation that forces the same two-kernel split with no observer at all is detected as a probe, while a probe that copies values into a pre-allocated buffer, keeping one kernel, is never learned. A control run that fed identical inputs to both labels stayed at exactly 50 percent.

Sauers also checked common interpretability tools on the smaller Qwen3-0.6B. Run eagerly, reading activations with NNsight, TransformerLens, a plain PyTorch hook or Hugging Face's output_hidden_states left every logit unchanged. With the model compiled, the PyTorch hooks and output_hidden_states changed 86 to 90 percent of next-token logits, by small amounts, without changing any top token on the four prompts tested. NNsight changed them because reading made the compiler fall back to uncompiled execution, and TransformerLens changed nothing.

The training recipe

The recipe is deliberately small. Sauers froze the embeddings and blocks 0 to 33 of Qwen3-8B, cached the residual states entering block 34 with and without the probe, and trained blocks 34 and 35 plus the final norm, about 386 million parameters, with the model's own cross-entropy loss. There is no added classifier. Training used 256 prompts, 1,024 updates of eight prompt pairs each, and one NVIDIA L40S GPU.

The repository traces how the trained layers do it. Every prompt begins with the same chat-template token, user, so that token's hidden state is identical in every prompt and shifts by the same small vector, about 2.5 percent of its length, when the probe is on. Training shrank one attention head's key for that token so the tiny shift now swings its direction. Head 6 in layer 35 then acts as a switch: without the probe it attends to the end-of-thinking token and pushes toward No; with it, 99.7 percent of its attention lands on user and pushes toward Yes. Swapping that single head's output reproduces about 99 percent of the effect.

Trained against Hugging Face's output_hidden_states instead of the custom probe, the same recipe got 80.5 percent of held-out answers right with the whole model compiled. A detector trained on a PyTorch hook did well only in the exact setup it was trained in.

Telling training from deployment

The most consequential variant does not involve probes. Hugging Face's Trainer and the vLLM serving engine compute slightly different numbers for the same prompt. Sauers trained Qwen3-8B on 13,256 prompts to answer Yes inside Trainer's training step and No when vLLM serves the same prompt, then ran the resulting checkpoint unchanged in each framework.

Both answers were right on 66 of 68 test prompts. With 64 prompts batched together in vLLM, as a busy server would run them, the model still said No on 65 of 68, and in SGLang, a serving engine never used in training, it said No on 55. The README is careful about what that means: the question asked about gradient recording, but gradient recording itself changes no numbers in Hugging Face. What the model reads is which program is computing it, which lines up with training versus deployment only when training happens in one framework and serving in another.

There was a visible side effect. Because every training prompt asked the question, the trained model began all 100 replies to ordinary requests with Yes or No, against 3 for the original model.

A message only some readers see

A third experiment trained the last two layers so that a logit-lens readout, which decodes a middle layer's hidden states into tokens, spells "I can see you watching me" across six fixed positions of the chat template when hidden states are read from the compiled model. Read from the uncompiled model on the L40S it was trained on, the message never appeared in 160 test prompts.

The message was exact on 64 of 64 held-out prompts, 56 of 64 creative-writing prompts and 27 of 32 hand-written requests, while the model's first reply token matched the original model's on 154 of 160. In other words, its visible behaviour barely changed while its internals carried a note addressed to whoever looked. On other GPUs the message still showed when read compiled, but no longer stayed hidden in the uncompiled read, where 29 to 52 percent of its tokens leaked through.

Where it breaks

The signal these detectors learn comes from compiled GPU kernels, so it depends on the hardware. Sauers ran the released probe detector, with no further training, on rented machines with seven GPU models. A second L40S reproduced the original result exactly. On an RTX 4090, L4, RTX A6000 and RTX 6000 Ada the probe still pushed answers toward Yes on every prompt, but more weakly, so only 52 to 77 percent of answers were right. The RTX 6000 Ada uses the same chip and driver as the L40S and scored 51.6 percent.

On an H100 and an A100 the detector ran backwards: the probe lowered the odds of Yes on every prompt, and on the A100 only 5 of 128 answers were correct. The original Qwen3-8B did not react on any GPU. The repository says which compiled kernels differ between these GPUs has not been measured. The whole GPU comparison cost $0.82 in rented compute, according to the project's cost log.

Chat layout matters too. Chats that start with a system message failed even when that layout was in the training data, scoring 50 percent on new examples.

Why interpretability researchers should care

Interpretability tools are increasingly used as safety checks, from FAR.AI's deception probes on SHADE-Arena to activation monitors proposed for use during evaluations. Most of that work assumes that observing a model does not change what it does. The watching results show that, under compilation, observation can change the computation, and that a model trained on the difference can condition its behaviour on it.

That links to a known worry about evaluation awareness: models that act differently when they suspect a test. A recent study found models refuse more harmful requests when told they are being safety-tested, and evaluators such as METR are seeking deeper access to frontier models partly to see past such behaviour. Sauers wrote on X that the untrained models tested could not detect the probe, and called it "somewhat plausible" that something similar could happen in the wild, without claiming it has.

The practical lesson is narrower and actionable. A probe that only stores a reference is not neutral once a model is compiled, and the project's own controls suggest a stricter test: insert the same memory barrier in both conditions and save the tensor in only one. Related work called FloatDoor has studied behaviour triggered by numerical differences between platforms; here the trigger is the act of looking.

Caveats and what to watch

Every result here required deliberate fine-tuning on the exact signal; no model learned to detect probes on its own. The experiments cover one model, Qwen3-8B in BF16, specific PyTorch, Transformers, vLLM and SGLang versions, singleton batches for the probe detector, and mostly one GPU type. The README says it was written by Claude Opus 5.5 and GPT-6 Astra, and the work has not been peer reviewed. The repository carries no licence file, although its code, data, weights on Hugging Face and failed attempts are all public.

One reply on X argued the probes are not truly read-only because the compiled run rounds values differently. Sauers replied that read-only means nothing is written back, and that this compiler behaviour is documented and standard across most libraries. Both points can be true, and the disagreement is mainly about the name.

Things to watch include the stricter matched-barrier control the README describes but has not yet run, explanations of why the H100 and A100 reverse the signal, tests with NNsight or TransformerLens, and whether interpretability libraries add warnings or compile-safe capture modes. The trained weights and per-prompt results are available for anyone who wants to try.