OpenAI cut ChatGPT storage read latency from 120 ms to 10 ms with caching and Rust

What OpenAI published

OpenAI on 23 October 2026 published the second part of an engineering series on Habitat, the internal storage platform behind ChatGPT, Codex, its API and internal services. The post, written by members of technical staff Jon Lee, Hrachya Hakobyan and Chaomin Yu, explains how the company scaled the database underneath Habitat, Microsoft's Azure Cosmos DB, to serve more than 1 billion users across 40 geographic regions.

The headline number is speed. OpenAI says the median time to read data from Habitat, measured globally, fell from roughly 120 milliseconds to roughly 10 milliseconds since it started focusing on performance earlier in 2026, a 12-fold improvement. It credits three changes: copying critical data into regions near users, an in-region cache, and rewriting the Habitat service from Python into Rust.

The post also reveals that OpenAI is building its own exabyte-scale online storage system for its bulkiest data, which would sit alongside, and for some workloads in place of, Azure Cosmos DB.

What Habitat is

Part I of the series, published on 11 September 2026, set out the basics. Habitat started as a small Python library that launched with GPTs at DevDay 2023, giving product teams a simple way to store and fetch data without dealing with schemas, routing, permissions or encryption. By the middle of 2025 it had become a standalone service, and OpenAI says it now handles more than 70 million requests per second and holds more than 500 petabytes of data.

Habitat deliberately does less than a normal database. It exposes a simple NoSQL interface built around objects and the links between them, modelled on Meta's TAO system, instead of letting teams write arbitrary SQL. OpenAI says this followed outages on its earlier Postgres setup, where a single expensive new query on a hot path could take the whole database down.

Part I also described the Rust rewrite. In the second quarter of 2026, two engineers using Codex and GPT-5.5 rewrote the whole service in Rust. That version now serves 95% of production requests, and OpenAI reports it is six times more CPU-efficient and 15 times more memory-efficient than the Python one. It is another example of AI-assisted ports to Rust, after projects such as Cloudflare and VoidZero's tsrs TypeScript compiler.

Three tries at sharing one database

Most of Part II is about multi-tenancy, meaning many products sharing one database instead of each team running its own. OpenAI describes going round in a circle: one shared Azure Cosmos DB container, then many single-tenant containers, then back to a shared container by default.

In the first phase, OpenAI relied on Cosmos DB's documented limit of 10,000 request units per second for each partition key to hold back heavy users. That broke down in three ways. Different products picked the same keys, such as user or workspace IDs, and so competed for one user's limit. Busy workloads on a niche feature could share a physical partition with login and signup, a classic noisy-neighbour problem. And as data grew, partitions split and each half got half the throughput, so storage growth alone could starve active workloads.

In the second phase, teams were told to create separate containers for critical services. That reduced site-wide outages but pushed capacity planning onto every product team. OpenAI says teams often shipped features without scaling their container first, and engineers ended up in incident rooms on launch day trying to add capacity at the last minute.

The third phase went back to a shared container, with fixes built together with the Azure team: throttling the hottest keys first, giving offline jobs such as bulk inserts a lower priority than interactive requests, and a new serverless mode in which every physical partition can scale to 10,000 request units per second and splits based on usage as well as storage. OpenAI says it has had no provisioning or noisy-neighbour incidents in the months since.

When a managed service stops being managed

OpenAI's largest Cosmos DB account stores tens of petabytes and serves more than 20 million queries per second. The company says that, to its knowledge, no other company has deployed Cosmos DB at that scale.

Two incidents showed the limits of treating it as an ordinary cloud service. In late 2024, a power outage hit the primary Azure region for one of OpenAI's largest accounts, and the team learned that its biggest accounts would take several hours to fail over to a backup region, something neither side had tested at that size. About a month later, moving those accounts back caused another serious outage: a quirk in the Cosmos DB client sent every connection back to the original region at once, overwhelming its gateway.

Today, OpenAI says, the two companies jointly monitor every layer from Habitat down to Cosmos DB's physical storage, test changes on a production-sized but non-production account, work on the client libraries together, and can page each other directly without going through support tiers.

How reads got 12 times faster

Reads make up more than 90% of Habitat's requests, so that is where OpenAI focused. Ideally, data would live near the user, but the post says that is limited by physical capacity: many regions cannot hold the storage their share of traffic would need, so OpenAI sometimes has to choose between storage close to users and more computing power for serving.

Instead it uses layers. The smallest and most latency-sensitive datasets, such as authentication metadata needed on every request, are copied widely, in some cases to all 40 Azure regions where OpenAI serves traffic. Everything else leans on a cache of recently used objects held in Valkey servers close to where requests are processed. The Rust rewrite then removed overhead in the service itself.

The change feed behind the cache

Caches usually force a trade-off between speed and freshness: keep data longer and you serve stale results, expire it quickly and the cache helps less. OpenAI says it avoids most of that by using change data capture, a stream of every write, from Cosmos DB's change feed. A service consumes the stream and deletes the matching cache entries worldwide within seconds, so the next read fetches the new version.

The same pipeline feeds search indexes, data movement and analytics. OpenAI says it carries millions of changes per second from thousands of containers through more than a hundred Kafka topics, and that it even had to change Azure's change-feed software to save its progress more efficiently at this scale.

OpenAI is building its own storage engine next

The most forward-looking detail is in the last section. To handle the next billion users, OpenAI says it is building a custom, in-house online storage system designed for exabytes, aimed at Habitat's bulkiest data types. It combines the RocksDB Cloud core from Rockset, the database startup OpenAI bought in 2024, with Raft-based replication, and implements the same document model as Cosmos DB.

The design separates storage from computing and copies data at the physical level, which OpenAI says needs less flash storage in exchange for a lower peak query rate. The company gave no timeline, no cost figures and no indication of how much data would move off Cosmos DB.

What the post does not say

This is OpenAI describing its own systems, and none of the figures has been independently checked. The latency chart shows medians only, and OpenAI acknowledges that the slowest requests, which still have to fetch uncached data from distant regions, remain a problem. It gives no tail-latency numbers, error rates or costs.

The post also leaves out how conversation storage, which it mentions separately, fits into Habitat, and how the in-house engine will handle data residency rules. For context on how open-source databases are coping with AI-era load in a different way, see why PostgreSQL 19 slipped to November.

Why it matters

For engineers, the series is a rare look inside one of the largest consumer AI services: how a small team kept a narrow storage interface, put up with Python longer than expected, and leaned on a cloud partner until outages forced much closer work. The lesson OpenAI draws is that, at enough scale, a managed service becomes something you operate together with the provider.

For everyone else, it explains part of why ChatGPT feels faster than it used to and hints at where OpenAI's infrastructure is heading: more of its own hardware and software, less dependence on a single provider's database. It also adds to the evidence that AI coding tools are now trusted with large rewrites of production systems, as in Theo Browne's tsc-rs experiment.