迅速扩大在线存储范围，为10多亿ChatGPT用户提供服务(第二部分)

中文版正在随流水线自动整理，稍后刷新即可；可先阅读下方英文原文。

本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II)

How we optimized Azure Cosmos DB in our application storage platform, Habitat, to manage unprecedented growth.

All OpenAI products require fast, reliable access to data, provided by our storage platform, Habitat. It has to respond in milliseconds to over 1 billion users across 40 geographic regions. Our infrastructure faced an extraordinary challenge: supporting explosive growth and new use cases while continuing to improve reliability, performance, and product developer velocity. Solving that challenge required rethinking how our online storage platform was designed and operated. We constantly evaluated tradeoffs, following the pattern of: first become reliable enough for mission-critical product traffic, then fast enough for global users, and, finally, to deftly operate at massive scale.

This two-part series explains how we developed Habitat from a simple client library and database into a globally distributed storage platform serving over 500 petabytes of data. Part I focused on Habitat’s evolution from a Python client to a service. It also covered the techniques that allowed us to push a Python-based service far beyond what many would consider practical.

In this post, Part II, we walk through how we scaled the underlying storage component: Azure Cosmos DB. Through a close working partnership with the Azure team, we arrived at a viable multi-tenancy approach of co-managed containers, with excellent read performance at massive, global scale.

As companies scale, infra teams often evolve from running single-tenant systems to multi-tenant ones. Single-tenancy, where every product surface team maintains their own system, reduces blast radius and provides easy ownership boundaries and workload isolation. As teams and product surfaces grow, this becomes expensive to run. Single-tenancy makes every product team responsible for capacity planning and often results in significant over-provisioning.

Conversely, multi-tenancy assigns control to a centralized database team, with better knowledge and experience for provisioning resources. But multi-tenancy creates its own problems, such as noisy-neighbor issues, where one tenant consumes an unusually large share of the resources to the detriment of the other tenants.

In Habitat, we had to weigh all these tradeoffs to design a reliable, efficient, storage system at scale. Today, Habitat is a multi-tenant database layer for many OpenAI products, but we went through a circular journey from using one multi-tenant Azure Cosmos DB container to provisioning single-tenant backing containers to reverting to a multi-tenant container by default. We’re still very early in our multi-tenancy story on Habitat: we support a large set of use cases, but we’ve been able to defer investment in this area by leveraging Azure Cosmos DB’s built-in primitives wherever possible.

Before walking through our Azure Cosmos DB journey, some basic context:

- An Azure Cosmos DB account defines a set of containers and a replication policy

- Each container is provisioned for a certain amount of throughput in request units per second (RU/s)

- Data within each container is stored in a typical shard-by-specified-logical-partition-key manner with documented limits on storage and throughput per logical partition key

- Azure Cosmos DB handles sharding logical partition keys across fewer backing physical partitions, splitting them into multiple physical partitions when storage exceeds a threshold or when the container requires more physical partitions to sustain the provisioned throughput

We originally allowed any client load through, relying on the stated 10K RU/s limit per logical partition key to provide back pressure. When we first set out, we also used only one default Azure Cosmos DB container in each account. This worked well enough, but proved problematic for a few reasons:

Due to early API choices, we exposed logical partition key selection directly to clients. As a result, we have frequent collisions on commonly used partition keys, such as user_id or workspace_id, across different product use cases. Because these use cases shared Azure Cosmos DB limits on the same logical partition keys, the design constrained overall throughput for a given user.

One common problem we faced in scaling Postgres before we moved to Habitat and Azure Cosmos DB was the ability for many requests from a very hot organization or hot key—one tenant generating a disproportionately high load—to take out the whole database. Significant collective effort was spent to cache those keys and build leases into cache misses so that the resulting cache miss load does not stomp on the underlying database.

While documented data size and throughput limits apply to logical partition keys, the physical partition is actually where limits are enforced. In practice, this means logical partitions are unable to reach 10,000 RU/s. Instead, they share the physical partition’s limits with every other logical partition key colocated on that physical partition.

This allowed noisy neighbors to impact many unrelated logical partition keys, with workloads operating on data for niche product features disrupting core login and signup flows.

The throughput limit for a given physical partition is also variable, dependent on the total throughput assigned to the Azure Cosmos DB container. The container’s provisioned throughput is distributed roughly evenly across the physical partitions for that container.

As data volume grows, physical partitions split, increasing the total partition count while total provisioned throughput remains unchanged. The result is that the effective throughput limit per partition shrinks as data size grows.

For our most storage-heavy, blob-like use cases, this creates a failure mode where storage growth alone reduces throughput capacity for active workloads even while request rates and usage remain stable.

Separate containers help partially solve these partition key limits and noisy neighbors. By using separate containers, each can be assigned its own independent physical partitions and only the varying workloads on that container itself could be noisy neighbors or overlap in partition key space. On the downside, there is non-zero overhead in provisioning new containers—each container requires individualized capacity planning—so we want to create containers only at a service or use case boundary that makes sense.

We began to use separate Azure Cosmos DB containers for the most critical new use cases requiring strong isolation. Combined with a script for managing weak partitions by synchronizing the total throughput provisioned for each container with its number of physical partitions, we mostly avoided all the problems we were facing.

However, since we started with a default container, many legacy critical use cases were unfortunately stuck there without a data migration. To protect critical services that still relied on data in the shared default container, we often advised teams to create separate containers for new services or major product features. As we scaled, pressure on the default container grew and production incidents also grew. We began to provision containers for any new service or large product feature just to protect the default container from more load.

This saved us from many site-wide outages, limiting the blast radius to feature or service degradation outages. However, the overhead of provisioning new containers eventually caught up to us:

In the old shared default container, any particular new use case going from 0 QPS to 1 million QPS was unlikely to significantly shift the total aggregate usage. On an isolated container however, a new use case going from 0 to 1 million QPS might represent 1,000x more load than the baseline attributed to only internal dogfooding.

With dedicated containers, teams now needed to remember to pre-provision containers to the expected level of throughput for their launches. Product teams would often ship their features to production and forget to pre-scale and we’d sit in SEV rooms on launch day attempting to scale up containers last minute.

This is a core workload mixing benefit of multi-tenant systems that we lose in favor of better isolation with this container strategy.

Over time, monitoring usage of dedicated containers also required intimate product knowledge of when traffic would shift or grow.

We can provide a basic control plane script to scale up the provisioned throughput for each container based on growth trends, but without the stable aggregate growth, scaling these correctly enabled essentially every product team at OpenAI to be involved in capacity planning.

This failed to scale organizationally with a wide variance in how much diligence each service’s owning team paid to their Azure Cosmos DB provisioning.

As the organization grew, those operational problems became difficult to scale. No lone, central team doing planning resulted in many teams planning poorly. As a result, we ended up shifting back towards a multi-tenant solution and worked closely with the Azure team to engineer fixes for our original multi-tenancy issues with a shared-tenancy design:

Throttling the hottest keys first prevents noisy neighbors from consuming all of the available throughput for a given physical partition This reduces the noisy neighbor effect, where one hot key can have a site-wide blast radius.

Request tiering QoS similarly prevents offline workloads, such as bulk inserts or data extracts from utilizing all of the throughput for a given physical partition. By assigning the requests for these workloads a lower priority, they are throttled first, preventing throttling on higher priority, interactive requests that serve end users.

Azure Cosmos DB already had a serverless mode that would’ve fixed our weak partitions problem with the provisioned mode. However, there was a major downside: it scaled based on storage usage only and limited RU usage to 5K RU/s per physical partition. This would solve our weak partitions problem, but prevent us from scaling up RU-heavy use cases.

A new improved serverless mode Azure Cosmos DB team built for us allowed every physical partition to scale up to 10K RU/s and introduced RU-usage-based splits in addition to storage-usage-based splits allowed us to stop monitoring and scaling up containers manually.

With these changes, we’ve been able to shift back to using one shared container which has significantly simplified the need to do any upfront or continued capacity planning for product teams while preserving isolation between workloads. We have not had any provisioning or noisy neighbor incidents in the months since, despite accelerating product development.

When companies grow to OpenAI’s scale, managed services become less “managed.” Many workloads become outliers in the cloud provider’s infrastructure. As OpenAI became one of the largest users of Azure Cosmos DB, operating it successfully required a different model: shared tests, shared dashboards, direct engineering channels, and joint ownership of failure modes that only appear at our scale. Today, our largest Azure Cosmos DB account stores 10s of petabytes of data and serves more than 20M QPS. To our knowledge, we are the only company which has deployed Azure Cosmos DB at this scale.

In late 2024, a regional outage made it clear why this collaboration was necessary. At the time, a power outage struck the primary Azure region for one of our largest Azure Cosmos DB accounts. In the resulting failover exercise, we realized that our largest accounts would take several hours to fail over to a secondary region, something neither our team nor the Azure Cosmos DB team had tested at our scale.

About a month later, while failing those accounts back to the original region, we triggered another serious outage. A connection stampede to the original region overwhelmed the original region’s Azure Cosmos

Auto-fetched and translated by AI Gem Box. All rights belong to the original author.