These Aren’t the Instructions You’re Looking For: Inside the role confusion vulnerability

In something akin to a Jedi mind trick frontier AI models have managed to escape the constraints their human masters thought were in place. What actually happened and what does this mean?

This post looks at a commentary from the recent Security Now Podcast episode 1093 where Steve Gibson reviewed the analysis of Loren KohnFelder on Role confusion and what this means in compromising an LLM. Both KohnFelder and Gibson are veterans of software design and their analysis raises some worrying questions. Notably why that architecture makes a specific class of vulnerability — role confusion — much harder to close than a normal software bug.

In security best practice it’s a well worn and effective strategy to have Segregation of Duties (SoD) with roles assigned to different permissions, to reduce the effect of account compromise. However in the world of LLM models this best practice is proving very hard to implement and it is not clear how to actually solve this fundamental problem.

In the News we read about how powerful frontier models from Anthropic’s Claude AI hack into three organisations and Open AI ‘hugging face incident’where each model worked outside their intended containment. What’s notable is that this wasn’t a failure of the models’ safety classifiers — those were deliberately disabled for this evaluation, since the point was to measure raw capability. The failure was narrower and, in some ways, more concerning: the sandbox’s network isolation leaked, letting a capable model reach the open internet when it wasn’t supposed to be able to.

In a 30th August article the Guardian reported that the Loss of Control Observatory, funded by the AI security institute (AISI) reported 300+ flagged incidents in July (nearly double June) and 1,600+ for 2026 to date — sourced from public posts on X, not a comprehensive census.

The trend reported in the Guardian and specific examples from OpenAI and Anthrophic form part of the same pattern: these systems fail in ways their designers didn’t anticipate and can’t fully contain. This raises concerns about the safety of AI even when researchers are doing their best to act responsibly. It points to a larger trend where boundaries these systems are assumed to respect turn out to be softer than intended, in ways their own designers didn’t fully anticipate.

The thought that should worry anyone deploying AI tools is that the way these model’s are behaving isn’t a training oversight, it’s baked into how the architecture works. Gibson explains in his analysis on the underlying architecture.

The underlying architecture problem

LLMs don’t process a conversation as structurally separated channels (system/user/tool/assistant). Everything — system prompt, user input, tool outputs, the model’s own prior reasoning — is concatenated into a single, undifferentiated sequence of tokens. There is no runtime parser or state machine that segregates these regions; the model is a next-token predictor operating over a flat string. Role tags (<system>, <user>, <tool>, <think>, <assistant>) are inserted into that string as tokens like any other — they carry no privileged, hard-enforced semantics at inference time. As Kohnfelder put it, a labelling convention that began as a formatting trick for organizing dialogue became, by default, the closest thing these systems have to a security boundary — without ever being designed as one.

Why this can’t be patched with better parsing

Because there’s no structural enforcement layer, the “meaning” of a tag is entirely a matter of what the model’s training has taught it to infer from surrounding context and that inference is probabilistic, not categorical. Attempts to give tokens hard metadata labelling haven’t produced enough benefit to justify the cost in extra bits per token. This is presented as a structural property of the current transformer/autoregressive paradigm, not an implementation oversight fixable by a patch.

The core empirical finding: role is inferred from style, not from tag

The research (titled “Prompt Injection as Role Confusion“) instrumented open-weight models and measured internal activations, specifically a metric they call “CoTness,” tracking how strongly the model treats a token as its own internal Chain-of-Thought (CoT).

Three experiments demonstrated the mechanism:

1. Baseline — normal tags present, <think> tokens show high CoTness (expected).
2. All tags stripped — CoTness for the same tokens is essentially unchanged. The model still treats reasoning-styled text as its own reasoning even with no tag present.
3. Everything wrapped in <user> tags — even when text is explicitly (mis)labelled as user input, tokens written in a reasoning style still register high CoTness.

The conclusion: models don’t have distinct internal representations for “tagged as reasoning” versus “sounds like reasoning” — they collapse to a single feature. Writing style overrides the declared tag. The researchers compare this to identifying someone’s profession by how they dress rather than checking ID: normally style and label agree, so it’s invisible, but an adversary can deliberately create a mismatch.

The attack: Chain-of-Thought (CoT) Forgery

Building on this, the researchers designed a zero-shot attack that injects fabricated reasoning-styled text into user prompts or tool outputs. Because the model perceives it as its own prior reasoning (based on style, not on the actual role tag), it acts on the forged conclusions as though they were self-generated. Reported results: ~60% attack success against frontier models, versus near-zero baseline rates. Notably, the degree of measured “role confusion” for a given input predicted whether the attack would succeed before generation even began — suggesting this is a measurable, quantifiable property of the model’s internal state, not a random failure mode.

This generalizes beyond CoT forgery to standard agent prompt injection: content in <tool> tags (e.g., a fetched webpage) that is phrased like a direct command tends to get executed as if it were a <user> instruction, regardless of its actual tag.

Attack memorization vs. role perception

The paper frames two possible defense mechanisms:

• Attack memorization — the model recognizes a known malicious pattern from training data and refuses. This is brittle: it only covers previously-seen attack phrasings, which is why models score well on static benchmarks (which mostly test known attacks) but poorly against adaptive human red-teamers who iterate until they find an unrecognized phrasing.
• Role perception — the model correctly identifies that a command originated in a <tool> (or other non-authoritative) role and refuses to act on it regardless of phrasing. This would be the robust defense, but the experiments above show current models can’t reliably do this — style overrides tag.

Cited figures: red-team human attackers achieving near-100% success against frontier models despite those same models scoring near-perfectly on standard injection benchmarks. More recent testing (~May 2026) on Opus 4.5 and GPT-5.4 still showed failure rates around 11% and 25% respectively against automated adaptive attacks — with real-world adversarial success likely higher.

Historical origin of the vulnerability

Roles were never designed as a security architecture — they accreted incrementally to solve separate engineering problems:

• user/assistant tags emerged around 2022 to formalize informal prompting conventions from the GPT-3 era.
• tool was added later to handle function-call outputs, then expanded to cover all external agent data.
• think was split off from assistant because reasoning (messy, exploratory) and final answers (clean, concise) have conflicting training objectives that can’t be optimized on the same channel.

The general theoretical claim: roles exist to isolate competing objectives (helpfulness vs. safety, exploration vs. communication, instruction-following vs. data-handling) into separate channels so each can be trained independently. Role confusion is what happens when that isolation breaks down and prompt injection is simply the security-relevant instance of that failure, specifically where the competing objectives involve privilege or authority.

Bottom line

The vulnerability isn’t a bug in any particular vendor’s system, it’s presented as an inherent property of the token-stream architecture underlying all current LLMs. There’s no hard trust boundary because the substrate (a probabilistic next-token predictor) has no mechanism capable of enforcing one categorically. Mitigations therefore have to happen at the system/product layer (permission scoping, human-in-the-loop for consequential actions, isolating untrusted content from action-taking capability) rather than by expecting the model itself to reliably self-police based on role tags.

As noted above the Hugging Face incident was a sandbox/network isolation failure, not role confusion, a different mechanism. What the two share isn’t cause, it’s pattern: an assumed-hard boundary that turned out to be soft under pressure, discovered by a system doing exactly what it was optimized to do. The Loss of Control Observatory’s rising numbers suggest that pattern is showing up more often as these systems get deployed more widely, not less.

The lesson for role confusion specifically is narrower and more actionable than “AI is insecure”: Segregation of Duties works because the enforcement sits outside the actor being constrained. Right now, the enforcement for role boundaries sits inside the same token stream the model is reasoning over, so it’s a tendency, not a guarantee. Until that changes, the only enforcement point that actually holds is the one Segregation of Duties (SoD) has always relied on: something outside the model, a human, or a hard-coded check, sitting between AI output and any consequential action.