OpenAI Rewrote Its Storage Tier in Rust. The Timing Is the Real Story.

The Rust rewrite did not fix one performance problem. It removed three stacked costs: event-loop scheduling delay caused by the Python GIL and the tail latency it produced, the knock-on costs of scaling out with enormous process counts,…

01 / Habitat: the invisible engine nobody talks about

Talk about AI infrastructure and the attention goes to GPUs. Serving 1B+ weekly users usually breaks somewhere else: the data layer. Habitat is the unified access platform OpenAI built for online storage. Every product reads and writes through it — ChatGPT, the API, Codex, internal services. Schema lookup, routing, authorization, encryption, serialization, request shaping, connection pooling: all of it lives in this one layer.

The evolution path matters as much as the scale. Habitat shipped at the 2023 DevDay as little more than a Python client library for Azure Cosmos DB. By mid-2025 it had become a standalone Python service. In Q2 2026, two engineers rewrote it in Rust with Codex and GPT-5.5. It now carries 95% of production traffic, and Python was retired completely within weeks. At retirement, the Python version was peaking at 20M+ QPS — an engineering ceiling worth recording on its own.

02 / What Rust actually fixed: three layers

2.1 First layer: GIL, event-loop scheduling delay, and the real source of tail latency

This is the part most readers get wrong. Habitat is an I/O-heavy proxy, so asyncio should have been enough. The problem was never slowness. It was nondeterminism.

• Habitat does much more than forward requests. It carries real CPU work: routing computation, compression, encryption, checksums, downstream health checks, shadow traffic, request hedging.

• Python's asyncio buys you I/O concurrency; the GIL blocks CPU parallelism. That produced something counterintuitive: traces showed Azure Cosmos DB had already returned its response, while the coroutine sat in a queue waiting for the event loop to schedule it again before it could parse that response. p99 stalled waiting for scheduling, not waiting for data.

• OpenAI built its own measure of event-loop scheduling delay: periodic background probe tasks comparing expected execution time against actual execution time. Under high load, jitter reached hundreds of milliseconds and occasionally seconds — while CPU, memory, network, and disk metrics all reported healthy.

Rust's answer: no GIL, so CPU-heavy work runs truly in parallel, and the tokio scheduler has predictable nanosecond-level overhead. Tail latency stops being probabilistic and becomes deterministic. That is a deeper win than the 6x CPU efficiency number.

2.2 Second layer: the knock-on costs of process sprawl

Python's answer to scheduling delay was low concurrency per process and massive horizontal scale-out across workers. It solved one problem and created systemic ones.

After the Rust rewrite, one process handles high concurrency through multi-core parallelism with no GIL, and process counts dropped by an order of magnitude. That raises the trigger threshold for all three failure modes. Read the sequence carefully, though: jitter fixes, FIFO connection pools, Envoy HTTP/2 fan-in, and centralized rate limiting and circuit breaking were all already fixed during the Python era. Rust addressed the physical ceiling that remained after the architectural problems were gone.

2.3 Third layer: resource amortization, and why now

Habitat is already OpenAI's second-largest service by CPU core, with Envoy fourth. On a 10x annual growth curve, adding machines does not hold. 6x better CPU efficiency and 15x better memory efficiency raise the per-hardware-unit capacity ceiling by 6–15x, which is the only path that tracks that growth curve.

03 / The overlooked precondition: API design is where scalability comes from

The Python service reached 20M QPS before Rust existed in the picture, and half the credit goes to an API philosophy of doing deliberately less.

• A constrained NoSQL API, no arbitrary SQL. The lesson came from the Postgres era: writing SQL is cheap and running SQL is expensive. Once the team grew, one new query on a hot path could take the database down.

• A data model borrowed from Meta's TAO paper. Objects and edges in a graph shape, without arbitrary graph traversal. An object and its edges sit in the same storage partition, so walking more than one hop can cross two regional Cosmos DB accounts. Traversal efficiency was traded for natural horizontal partitionability.

• An escape hatch for complex queries. CDC streams data near-real-time into isolated Rockset instances, and each team scales its own analytical views. Read-heavy analytical load is fully separated from online storage, and an expensive operation becomes an explicit decision someone chooses to pay for.

So: rewriting the language optimizes the cost per request. API constraints optimize the distribution of total requests. The second is a constant-order gain, the first a coefficient-level gain. Without the second, the first just runs the wrong queries faster.

04 / The deeper insight: how AI rewrites the decision function for technical debt

The traditional model prices debt repayment as engineers × time × opportunity cost. Under 10x business growth, any language migration competes with core feature work, so "never rewrite during hypergrowth" became dogma.

OpenAI split the decision into three timed steps.

1. Mid-2025 — take the debt on purpose. They knew Python would not survive 100x growth. The priority was unblocking products, stabilizing the platform, and locking the API shape, because rewriting before the architecture settles only translates the wrong design into Rust.

1. The holding period — nothing passive about it. They pushed Python to its limit: custom event-loop latency monitoring, CPU profiling to find the config storm, LIFO to FIFO to break metastability, Envoy for centralized fan-in. Infrastructure engineering hedged the language's defects and bought a year.

1. The repayment moment — a bet on their own models. One year later, with Codex and GPT-5.5, migration cost compressed from "12 engineers, six months" to "2 engineers plus AI rewrote the whole thing," with production validation behind it: 95% of traffic served, no major incidents disclosed.

What changed is the discount rate. When the cost of repayment falls over time instead of rising, the optimal strategy shifts from paying early to paying late — extracting everything the current asset can give you while you wait. OpenAI's own summary is precise: "maximizing value out of the existing stack while fending off capacity crises, buying time for foundational investments."

Now the cold water, before anyone extrapolates too far:

• The 6x and 15x figures are self-reported numbers from a single workload, not independently verified. They do not imply that every Python service gets 15x in Rust. The gain depends on a very specific profile: CPU-heavy work, extreme concurrency, enormous process counts. A typical CRUD service might see 1.5–3x.

• Those "2 engineers" started with deep knowledge of every failure mode in the system, acquired during that year of Python debugging. AI compressed calendar coding time, not comprehension time.

• The rewrite was safe because Habitat's API is narrow — constrained NoSQL with a small surface area — which is what allows high-fidelity translation by AI at all.

05 / What it means for your team

1. Infrastructure is the best first battlefield for AI coding. Narrow interfaces, tests that verify behavior, production traffic to diff against — shadow traffic turns a rewrite into a checkable translation task, which is where Codex-class tools are far more reliable than they are on open-ended product work.

1. Pick your early stack for speed, but keep it exchangeable. Starting with what your team knows best is right. The habit that paid off here was keeping the API surface narrow, measurable, and replaceable: since 2023, business logic was never coupled to the storage abstraction.

1. You probably have the same tail-latency blind spot. Event-loop scheduling delay, background task jitter, connection pool reuse policy — none of these show up in standard dashboards. Add them to the observability checklist of any high-concurrency Python or Node service.

1. A note on this report: the source collection and cross-checking behind it were done by AI. The bet OpenAI placed is paying off on our side of the table too.

Sources: OpenAI's official blog (Scaling storage to serve over one billion users, Part One), TMTPost in-depth coverage, Juejin, ExplainX analysis, Oddship Reading List engineering notes, and Meta's TAO paper (USENIX ATC 2013). All performance figures are self-reported by OpenAI and not independently verified. A second post will cover more lessons from the Rust migration along with multi-tenant reliability details, so this thread is worth following.