OpenAI’s Habitat shows why bounded work matters at scale

OpenAI’s account of Habitat, the online storage platform behind ChatGPT, Codex and other products, points to a useful operating decision: audit how much work one request can create, and where that work can wait, before approving more capacity or a language rewrite. Its largest numbers attract attention, but the transferable evidence is in the controls and failure mechanisms.

The OpenAI News RSS feed timestamps the article at 10:00 UTC on 11 September 2026. OpenAI’s current engineering post says Habitat serves products used by more than one billion people each week, handles more than 70 million requests per second across almost 40 geographic regions and holds more than 500 petabytes of data. Those are company-reported production figures; Index Us has not independently verified them.

There is also a live source discrepancy. The RSS item’s description says 22 million requests per second, while the current article says more than 70 million. The feed does not explain whether its summary is stale, refers to an earlier point in the system’s growth or is simply wrong. This analysis therefore does not treat either throughput figure as an independently established launch metric. The underlying engineering account remains useful because its mechanisms do not depend on choosing one number.

Start with a request budget

Habitat began as a Python client library connected to Azure Cosmos DB. OpenAI says that by mid-2025, coordinated updates across dozens of services had become brittle. Moving the logic into a standalone service centralised deployment, observability, routing, access controls and storage policy. From an operator’s perspective, the important change was the new choke point: storage behaviour could be controlled in one service rather than coordinated through library updates across product code.

The service exposes a constrained NoSQL interface for client-defined objects and edges. It does not accept arbitrary SQL, complex joins or unrestricted graph traversal. An object and its outgoing edges share a storage-level partition, although the remote object named by an edge may be in another account or region. Complex analytical and search access goes through change data capture into separate Rockset instances owned and scaled by client teams.

This gives the online path a more predictable request budget, at the cost of expressiveness and extra responsibility for product teams. Some graph operations become inefficient by design. The choice only works when those restrictions match the service’s real access patterns.

Two independent sources help define that trade-off without validating Habitat. The 2013 TAO paper, which OpenAI cites as an inspiration, describes objects, associations, a fixed query set, bounded range calls and sharding that maps a request to one cache server. Current Azure Cosmos DB partitioning documentation explains the corresponding database risk: uneven request distribution can create hot partitions, while a key that does not match read patterns can force cross-partition work. OpenAI has not published enough configuration detail for an external audit of its partition choices. The useful review question is whether the API contract and partition key bound the same work.

Look for waiting that CPU averages hide

OpenAI says one user action can generate hundreds of database calls. In that shape of system, one delayed call can become visible to the user. Google Research’s The Tail at Scale gives the broader context: growth and higher utilisation make temporary latency outliers increasingly important to the response time of a large interactive service.

Habitat’s Python traces reportedly showed that the database was not always where the request waited. asyncio allowed I/O to overlap, but routing, compression, encryption, checksumming, health checks, shadowing and hedging still consumed CPU. OpenAI says some high-percentile requests waited for their coroutine to be scheduled again even after the storage response was ready.

The team measured scheduler delay by comparing when periodic work should have run with when it actually ran. OpenAI reports jitter of hundreds of milliseconds at high utilisation and seconds in some edge cases. It kept concurrency low in each process and increased the worker-process count instead.

That response exposed another source of delay. A feature-flag client refreshed a large configuration every minute without jitter, across as many as eight Python processes in each pod. OpenAI says the synchronised JSON parsing stalled all workers. Targeting a smaller configuration, extending the refresh interval and adding jitter removed that identified spike. Aggregate CPU alone would not have explained either problem. Scheduler delay and coordinated background work need their own measurements.

Trace feedback, not just slow instances

The next failure was not a uniformly overloaded fleet. OpenAI says client-side pooling left some Habitat processes handling five to ten times the average concurrency. After a burst, a subset continued to degrade after the originating client had stopped.

Its explanation is a feedback loop in aiohttp connection reuse. A slow process returned its connection later; last-in, first-out reuse then chose that recently returned connection for more work. Capping connection reuse duration reportedly limited the degradation. OpenAI says a first-in, first-out patch broke the loop and reduced steady-state request variance, and that Habitat now relies mainly on Istio and Envoy for pooling and server-load-aware balancing.

A separate Meta engineering account from 2014 documents a related mechanism in a different system. Its most-recently-used pool kept selecting connections returned over a congested link, preserving the congestion after the initiating event. Meta adopted a least-recently-used policy with a maximum connection age, then repeatedly triggered both the old and new behaviours.

The common lesson is diagnostic, rather than a universal preference for FIFO. Google’s SRE guidance on cascading failures describes overload as positive feedback: slower work raises in-flight requests and resource use, while retries or redistribution can intensify the problem. It also describes cases where LIFO request queues can favour work that is still useful. Queue and reuse policies therefore need to be tested against the actual work and feedback path. Realistic overload tests, early rejection and small queues may help, but only where the traffic shape supports them.

One fix can spend another capacity budget

Reducing concurrency within each Python process increased the number of processes, which then multiplied downstream connections. OpenAI says deployments could produce CPU churn as connections cycled, and a connection leak could exhaust a network address translation gateway. Habitat used Envoy to combine HTTP/1 connections from those processes, multiplex upstream traffic over HTTP/2, and apply central rate limits and circuit breakers.

The current Envoy pooling documentation says HTTP/2 pools can carry concurrent requests over one connection, subject to configured limits. Its circuit-breaking documentation covers limits for connections, pending and active requests, retries and concurrent pools. These are descriptions of available controls. They do not disclose OpenAI’s configuration or demonstrate how well it worked.

This is a practical warning for much smaller services. Process count, process concurrency, open connections and downstream capacity are separate budgets. A change that protects the scheduler can consume connection or dependency capacity instead. Tests should include routine deployments, restart waves, leaked connections and retry storms while the dependency is near its limit, not just steady throughput on a healthy system.

Ask for matched evidence before rewriting

OpenAI says two engineers, using Codex and GPT-5.5, rewrote Habitat in Rust in the second quarter of 2026. The company reports that the Rust service now handles 95 per cent of production requests, uses CPU six times as efficiently and memory 15 times as efficiently as the Python version, and has lower average and tail latency.

The post does not provide the matched workload, hardware baseline, benchmark method or underlying measurements needed to reproduce those comparisons. The figures are OpenAI’s internal results, not a forecast for another service. More useful is the reported sequence: stabilise the service boundary, identify where time and resources are going, address the more urgent feedback risks, then compare implementations against the same accepted workload.

A review before approving a rewrite can be short, but it should cover the whole request path:

1. Enumerate request shapes and reject or isolate work with unbounded fan-out.
2. Measure scheduler delay, queue time and per-instance concurrency alongside aggregate CPU and memory.
3. Slow one destination and observe connection reuse, load balancing, retries and health checks.
4. Exercise deployments and recovery near dependency limits, with circuit breakers and load shedding visible.
5. Review partition keys against access patterns; even distribution and efficient reads may require different choices.

This analysis is based on public documentation. Index Us did not access Habitat, inspect OpenAI’s configuration, reproduce its failures or independently verify its scale and efficiency figures. Independent sources establish that bounded queries, tail latency, hot partitions and feedback failures are established design concerns; they do not verify OpenAI’s account. The immediate action is to audit one latency-sensitive service for unbounded work, hidden waiting and self-reinforcing load before treating a rewrite or more capacity as the primary fix.

Sources

1. OpenAI — News RSS
2. OpenAI — Rapidly scaling online storage
3. Google Research — The Tail at Scale
4. Google SRE — Addressing Cascading Failures
5. Meta Engineering — Solving the Mystery of Link Imbalance
6. USENIX — TAO distributed graph store paper
7. Envoy — Connection pooling
8. Envoy — Circuit breaking
9. Microsoft Learn — Azure Cosmos DB partitioning

Last reviewed: 13 Sept 2026. See our editorial policy for how we test, source and correct our work.