AWS open-sources Infinity, a runtime that fits 75,000 idle AI agents in 8GB

AWS releases Infinity

Amazon Web Services has open-sourced Infinity, an experimental runtime for AI agents that is built around one idea: an agent that is waiting for a tool should cost nothing. AWS announced it on its Open Source Blog on 23 October 2026, in a post by Shadaj Laddad, and published the code in the hydro-project/infinity repository on GitHub under the Apache 2.0 licence.

Infinity is part of Hydro, a research collaboration between AWS and UC Berkeley. AWS is careful to call it an early research release shared to get feedback, not a managed service or a supported product. The headline number in the post is that 75,000 idle agents fit in the 8 GB of memory of a Raspberry Pi 5. That figure is AWS's own estimate from its benchmark, and it comes with conditions explained below.

The problem: agents that sit and wait

AWS starts from a simple observation. Agents spend much of their time waiting on tools. A coding agent may start a build that runs for twenty minutes, a triage agent may wait hours for the next pull request, and an assistant may wait overnight for a human approval.

Most agent harnesses run the loop as blocking steps: call the model, call a tool, wait for the tool, repeat. While the agent waits, it holds a process and its memory, and it cannot react to anything that happens in the meantime. Run thousands of long-lived agents this way and most of the bill is for waiting.

This is the same pressure behind other recent agent infrastructure launches, such as Vercel's open-source durable harness for coding agents and Google Cloud's Agent Compute Engine, which each tackle a different part of keeping long-running agents alive and contained.

Agents as stackless coroutines

Infinity borrows its central trick from programming languages. In Rust, JavaScript or Python, an async function that awaits a network request is suspended, and the thread moves on. Rust implements this with stackless coroutines: the compiler turns each function into a small state machine that keeps only the values needed at each await point, instead of a full call stack.

AWS argues an LLM agent can be suspended the same way. At the moment an agent calls a tool, its conversation history already holds everything needed to continue. If the runtime saves that history, it can throw away all in-memory state, and any process with access to the store can pick the agent up later.

So Infinity runs each agent as a series of bounded slices. A slice loads the history, runs one model completion, saves the output and dispatches at most one tool call. Then it ends. Tool calls act as await points, and a slice never blocks on a tool, a timer or any other outside operation. Anything that can wake an agent, including user text and tool results, arrives as a message on an input queue.

For developers, the visible change is how tools are written. A tool's execute method starts the work and returns at once. When the work finishes, the tool sends its result to the agent's input queue, whether that takes a hundred milliseconds or three days.

75,000 agents in 8 GB

Because a suspended agent is just its saved history, keeping many agents resident costs roughly the size of their histories. AWS tested this with a benchmark using a simulated model, not a real one. Each agent ran twenty turns, and each turn streamed a completion of about 1.2 KB that called a tool, received the result through the queue, and finished with a second completion.

The run used 80,000 agents and 3.2 million completions on a single thread of an AMD EPYC 9R14 server, with in-memory stores. AWS says resident memory grew linearly at about 100 KB per idle agent. At that rate, about 75,000 agents would fit in 8 GB, which is where the Raspberry Pi comparison comes from.

The comparison is an extrapolation from a server benchmark, not a test run on a Pi. Real agents with long histories, large tool outputs or images would use more memory each, and model calls, not memory, are usually the main cost of running agents.

Running on Lambda with nothing idle

The second payoff is serverless deployment. Since a slice never waits and loads all its state from storage, it can run in a function that starts, does one step and exits. Infinity calls this step mode, and the same tools and model providers work in both the local and the serverless mode.

AWS's reference deployment uses an SQS FIFO queue as the input queue, with one message group per agent thread. Each delivery invokes an AWS Lambda function that runs one step and exits. Conversation histories live in Aurora DSQL, deduplication records in DynamoDB, and timers use SQS delivery delays and EventBridge Scheduler. The repository includes AWS CDK constructs that set this up.

The post lists what this requires: bounded execution time per slice, no attachment to any one process, idempotent handling because queues can deliver a message twice, turns that are only saved once complete, and per-thread ordering enforced by the FIFO queue rather than distributed locks. The result, in AWS's words, is that an agent waiting on a three-day CI pipeline uses no compute until the pipeline reports back.

Synthetic tool calls for events

The most interesting design problem in the post is how to tell a model about events. Model APIs expect each tool call to get exactly one result. But a long-running command produces output over time, and a subscription to a repository should produce an event every time a pull request is opened.

AWS says it rejected the obvious options. A second result for the same call breaks the API's rules. Posting the event as a user message makes the model think the user said it. Putting it in the system prompt loses its place in the timeline.

Instead, Infinity inserts a synthetic tool call the model never made, named receive_event__injected, followed by a result holding the event. Its arguments point back to the original call, so the model knows which of its requests the event belongs to. The runtime never offers that name as a real tool and rejects it if a model tries to call it.

Child agents without a message bus

The same mechanism handles teams of agents. Every agent gets a built-in spawn_thread tool that creates a child with its own queue. The call returns as soon as the child exists, the parent carries on, and the child's reports arrive later as synthetic events tied to that spawn call.

By default, each incoming event is handled by a short-lived child thread that inherits the parent's context, so a busy subscription does not flood the parent's context window. A tool can instead mark an event as associative, which adds it straight to the parent's history. That suits updates the agent is actively following, such as output from a test run it started.

RAP and Infinity Code

Infinity ships with a tool protocol of its own. The Reactive Agent Protocol, or RAP, makes tool calls fire-and-forget over HTTP: the tool acknowledges at once, the agent can shut down, and the tool posts the result to a callback URL when it is done. The documentation says existing MCP servers work as RAP tools through a compatibility layer. Our explainer on MCP tools and resources covers the synchronous model RAP is reacting to.

The project also includes Infinity Code, a local coding agent built on the runtime. It works in a sandboxed copy of the repository, as a Jujutsu workspace or git worktree, runs builds and tests in the background, and hands back each change as a diff for review. The documented setup installs it with Cargo and uses Amazon Bedrock as the model provider with the user's AWS credentials.

What it does not prove yet

AWS lists open questions itself. Models are mostly trained on conversations where tool calls are synchronous, so it is not yet clear how well they reason about many interleaved events. The scheduler currently runs threads in arrival order, without priorities for user-facing work. Compaction for large, long-running trees of agents has not been studied.

The security question is the one to watch. AWS asks how a runtime should establish the provenance of events when agents exposed to the internet may receive inauthentic payloads, without giving up the stateless design. An agent that wakes up on webhook events is only as safe as its checks on who sent them, and that is still listed as an open problem.

The project is also young in public terms. The repository was created in April 2026 and had about two dozen stars when AI Lookout checked it on 24 October, with no independent benchmarks yet.

Who should look at it

Infinity is most relevant to teams running many long-lived, event-driven agents, such as repository watchers, approval workflows or CI helpers, where idle time dominates. It is written in Rust, and the serverless path is built around AWS services, though the queue and stores are pluggable. AWS's earlier Strands sandbox work points in a similar direction of agent infrastructure built from its existing cloud pieces.

For everyone else, the useful takeaway is the pattern. Treating each tool call as a suspension point and the conversation history as the saved state is a clean way to think about agents that wait. Whether the synthetic-event trick holds up across models is the part that needs outside testing.