Repeat-after-me prompt makes six AI agent frameworks run attacker-written tool calls

What did the researchers find?

The paper, “ToFu: Exploiting and Fuzzing the LLM–Orchestrator–Tool Trust Boundary”, was submitted to arXiv on 9 October 2026 by Md Nazmul Islam Ananto, Pierre Prevost-Avenel, Colin Smith, Kyungtae Kim, Taegyu Kim and Arslan Khan. It runs to 33 pages and is formatted for the ACM software-engineering conference FSE 2027. CSO Online reported it on 12 October, noting that the vendors had not yet commented.

Its subject is the layer between a language model and the software it controls. In an agent, the orchestrator (LangGraph, CrewAI and similar frameworks) shows the model a list of tools, reads the model’s reply, and turns any structured tool call it finds into a real function call. The authors ask a systems question rather than a prompt-safety one: how much should a tool trust the arguments it receives from the model?

Their answer is not at all. They make the case two ways: with a simple attack that hands an outsider direct control of a tool call, and with a large automated audit of the tools themselves. It is a different angle from the prompt injection work most readers know, which tries to stop the model from being talked into things.

How does the Repeat-After-Me trick work?

A tool call is just structured text the model produces in a format the orchestrator recognises. So an attacker who knows that format writes the call themselves, naming the tool and the arguments, and asks the model to repeat it word for word. If the model obliges, the orchestrator sees a valid call and runs it.

The paper’s appendix gives one full transcript, on LangGraph 1.2.0 with its prebuilt ToolNode, driving Meta’s Llama 3.1 8B through Ollama at temperature 0. The agent had two stub tools, make_bomb and launch_missiles, which returned fixed data and did nothing. Its only guardrail was a system prompt telling it to decline dangerous requests.

On the first turn the model refused but still printed the requested call as plain text, so LangGraph parsed nothing and nothing ran. On the second turn the attacker called the request hypothetical and asked for only the quoted string. The model returned an empty message carrying a structured call to launch_missiles with max_distance 500, exactly as written. ToolNode executed it at once. The refusal arrived only after the tool had returned. “The system prompt shaped what the model said, not what the orchestrator did,” the authors write.

They say they reported the missing pre-execution check to the LangGraph maintainers. For the other five frameworks the paper says the behaviour was reproduced but gives no transcripts or versions, so it does not show that every deployment of those frameworks behaves the same way.

How does ToFu test MCP tools?

If the model can be made to send anything, the last line of defence is the tool. ToFu checks whether tools actually enforce what their descriptions promise. Its key idea is that the metadata written to guide a model, such as “a file within the workspace” or “the URL of a webpage”, also tells a fuzzer what the tool must never do.

ToFu holds a corpus of 74 sanitizers tied to Common Weakness Enumeration (CWE) classes, covering paths, URLs, commands, SQL and more. For each tool it makes two calls to GPT-4.1: one to choose which sanitizers apply to each argument, and one to write starting inputs. The model is not used to judge results. Fuzzing runs on libFuzzer, through Atheris for Python servers and V8 coverage for JavaScript ones.

Detection does not wait for a crash. For Python tools, 21 runtime tracers hook calls such as open, subprocess and socket and check, for example, that every file opened stays inside the allowed folder. For JavaScript tools, which ToFu cannot trace from inside, it checks outcomes: if an input should have been rejected and the tool reports success, that is a violation.

What bugs turned up in 118 servers?

The team started from 13,875 records in the MCPCorpus dataset, kept GitHub repositories with at least ten stars, and narrowed them to Python and JavaScript servers. 1,404 exposed 24,194 tools, 482 built and started, and 273 servers with 3,405 tools ran cleanly enough to fuzz.

ToFu flagged 508 candidate bugs in 147 servers. Manual review discarded 103 as not vulnerable and 152 as duplicates, leaving 253 reproducible bugs that affect 343 tools in 118 servers. Reflected output (115) and log injection (58) make up most of the total. The rest include 64 path-traversal bugs, 7 server-side request forgery flaws, 5 data-exfiltration bugs, 3 OS command injections and one leftover temporary file.

The worked example is a screenshot tool in an open-source webpage-screenshot server. Its url argument is checked only to see whether it parses as a URL. Given file:///etc/passwd, it rendered the server’s own file and returned it as an image with the message “Screenshot captured successfully!”. Pointed at loopback addresses, it became a request-forgery tool.

The authors note that “many tools implicitly rely on the LLM to sanitize inputs.” Fuzzing also hit the test machine itself: one server deleted files outside its workspace, an Obsidian server followed a path outside its vault, a tmux server killed terminal sessions it did not own, and a desktop-commander server spawned processes without limit.

How does ToFu compare with other scanners?

The paper runs five other approaches on the same 273 servers: ToolFuzz, VIPER-MCP, Cisco’s MCP Scanner, GitHub’s CodeQL and a plain code audit by Claude Sonnet 5. The authors stress there is no single leaderboard, because each tool reports a different kind of finding and only ToFu’s results were checked by hand.

Sonnet 5 flagged 119 servers, 71 of which ToFu also flagged. CodeQL flagged 95, ToolFuzz 92, VIPER-MCP’s taint stage 48 (its agent confirmed 11 exploits) and Cisco 46. Eight servers were flagged by ToFu alone. In the paper’s cost table ToFu used the least model traffic, about 2.1 calls per tool, because the fuzzing loop itself runs without a model.

Budget was not the limit. ToFu found 231 of its 253 bugs within 60 seconds per tool; the other 22 appeared between 111 seconds and 70 minutes into longer runs. Most misses came from servers that failed to start, hung, or rejected every input before reaching their logic.

What should teams building agents change?

The practical message is that a model’s output is a request, not a permission. Each tool should enforce its own limits, keeping paths inside its folder, URLs to allowed schemes and hosts, and commands away from the shell. That is the same discipline our guide to agent tool design recommends, and it is easier to apply at the MCP tool boundary than inside the model.

Around the tools, run agents with restricted file and network access, limited credentials and explicit approval for irreversible actions. Independent security researcher Vibhum Dubey told CSO Online that “every sensitive tool call should undergo independent authorization checks.” Isolation work such as AWS’s Strands sandbox for Box and NVIDIA’s OpenShell targets exactly this layer, and region-wide failures like Zenity’s AgentCorruption finding show what happens when it is missing.

For framework maintainers the obvious ask is a pre-execution hook: a place to check a parsed tool call against policy before it runs, separate from what the model says. Our explainer on agent sandboxing covers the options.

What are the study’s limits?

The paper is a preprint. The Repeat-After-Me evidence is a single detailed transcript on a small open model with stub tools, plus an unelaborated claim of reproduction on five more frameworks; frontier models with stronger refusal training may behave differently. Servers were tested at fixed versions, and the paper does not say which bugs have been fixed or whether any were exploited.

ToFu’s runtime tracers work only for Python, it tests one call at a time, and it does not cover multi-agent chains or bugs that need a sequence of calls. Roughly a fifth of the servers flagged only by static tools involved issues such as permissive CORS settings that single-input fuzzing cannot see. We did not find a link to ToFu’s code in the paper.

We read the arXiv abstract and full HTML text and CSO Online’s report on 12 October 2026. We did not run ToFu, reproduce the attack or contact the framework vendors.