OpenAI 缩放吉特
在投资吉特生态系统的同时，建立代理规模的源控制。
在过去的一年里，对我们的单一财产的规模和要求以惊人的速度增加。 显然，我们需要修改我们的源控制系统，以达到Codex所允许的并行工作水平。
我们建造了Pando,Git仓库的新后台，没有中断任何工程工作，也没有改变我们团队使用Git的方式。 Pando现在将所有开发都用在了OpenAI的单列波上。 Pando旨在实现两个雄心勃勃的目标：第一，横向可扩展的写作，优化，以便同时向许多分支推进。 其二，使阅读的吞吐量最大化，因此客户端可以快速地下载数据以充分饱和其可用的网络带宽。
我们实现了这些目标，同时保持了与Git协议和客户端的相容性，使我们的工程师和agents都能将Pando无缝地引入他们现有的工作流程。 在生产中，潘多将中位推延性削减了近5×. 它现在每天提供几乎一百万伏特的数据。
以及为何我们选择与Git保持兼容。 描述为什么潘多能满足OpenAI的可缩放需要。 以及我们如何继续为Git捐款
Codex改变了OpenAI的开发规模和速度，使工程师能够平行地做更多的工作。 这意味着更多的分支和推动，更多的合并，以及更多的从发展环境，研究工作，和CI中获取和复制。
这些需要复合体。 一个工程师现在有数个变化可以同时为CI准备，每个推力可以设置几个任务来获取同样的承诺。
从去年年底开始支持我们独身的系统一直难以跟上我们遭受了缓慢的追赶和推进伴随着失败的合并，可靠性有时低于90%。 我们是否达到了Git本身的极限，还是我们为独身服务的方式的极限？Git是从一个通过电子邮件与内核交换补丁的社区中成长而来。 二十多年来，随着软件的构建方式的改变，开发者在Git周围对工具进行了改造。 当我们开始在自己的单簧管中感受到摩擦时，重要的是，我们可以继续推进这项工作。
我们在重建后端的同时，找到了一种方法来保持这种熟悉的工作流程，同时处理向不同分支的推力，同时为许多读者提供同样的历史。 以Git作为我们源控工作流程的基础，意味着对客户端的改进在OpenAI之外可能有用。 我们想促进这些改进，从我们复制的Git(在新窗口打开)到上游，与更广泛的社区分享。
缩放源控制系统有两个基本维度：许多作家向不同的分支推来推去，许多读者获取相同的历史。 潘道为两者优化。
首先，潘多独立地协调每个分支的更新。 对不相关的分支的更新可以平行进行，取取分支的读者可以获得该修改所需的历史。 取舍是，一个看几个分支的读者可以在不同的时间点看到它们。
第二，Pando重用包文件(Git用于将多个对象压缩在一起的格式)，而不是为每个客户端生成单个包文件。 一个客户端可能会下载一些它已经拥有的物件，但服务器可以向许多客户端提供相同的字节，而计算很少。
这些选择让我们跨分支书写，服务从缓存读取。 让我们从Git如何储存和转移历史开始。
Git必须能够保持寄存器的两个部分同步：对象和引用。
一个 Git 寄存器记录了文件的快照和连接它们的历史作为对象的图。 物体可以是blob、树、承诺或标记。 Blobs持有一个单个文件的内容，而树名与blobs或其他树相关联，形成目录结构。 指定根树和0个或更多父承诺，构成您的寄存器的历史。
物体由其类型和内容的散列物识别。 对象也是不可改变的：记录一个已更改的文件会创建一个新的blob，而没有改变的文件和目录可以再使用已有的对象。
分行和标签是参考文献，或指 " ref "。 承诺建立一个分支
Git 存储物体处于“放荡”或“包装”状态。 放放对象被存储在其对象ID识别的单个文件中。 被包装的物体被一起储存在一个包文件里，使得类似的物体能够作为三角形互相存储。
在基特语中，如果基特能够通过通过父母、树和blobs来跟踪犯罪的联系来发现它，那么一个物体就能够从犯罪中达到。 在获取时，客户端告诉服务器它想要的以及它已经拥有的。 服务器发现从通缉犯中可以到达的对象，排除那些它知道客户端已经拥有的对象，并将其余的以同样格式的 Git 格式发送到磁盘上。 一个完整的克隆开始没有任何对象，所以它需要从被克隆的历史中可以到达的所有对象。
通过要求服务器更新一个或多个 refs 来将对象推向另一个方向。 客户端使用服务器的一组已公布的参考文献来避免发送服务器已经拥有的对象。 因此被发送的包可以依赖于现有的对象：仅接收包不足以重建新的历史。
复制一个Git寄存器意味着在多个服务器上保留对象和引用。 在复制品能够服务于参考物的新价值之前，它必须能够提供从该价值中可以到达的物体。 服务还需要协调相竞的参考更新，即推动试图将同一个分支从同个旧尖端移动到不同的新承诺。 将所有写作都强制一个单一顺序的系统到寄存器中，可以使被推向无关的分支互相等待。
在设计Pando时，我们用Git评估了常规的源控制方法：使用三相传输(3PC，协调多个节点的承诺的协议)来协调存储器复制器的更新
在我们模拟的三相承诺设计中，Git服务保存了多份复制品的完整寄存器。 当任何服务器收到一推时，收到的包文件会被复制并同时发送给所有复制品。 要发布推文，大多数复制品必须接受参考更新。 管理此功能的服务器是协调器，它需要协调器和相互复制的多轮通信。
为了接受参考更新，每本复制件会将它仍然指向客户端期望的承诺的参考副本锁定并检查。 协调员随后向接受的复制品发出承诺前信息。 一旦多数人承认承诺前的信息，协调员就让他们进行更新。 任何最新的复制品都可以从存储器的本地拷贝中为新的分支提示做广告。
保存完整的复制品可以在当地提供阅读服务，并保证客户无论选择何种复制品，都能对所有参考文献有一致的看法。 因此，参考文献一次更新一次，每次推后都需要多次往返来协调翻版的参考文献更新。
等待法定人数使复制磁盘的性能和通信的延迟进入了推动路径。 随着写作量的增加，复制品必须跟上接收到的包和阅读流量。
在基于WAL的我们所模拟的设计中，Git服务使用blob存储中共享的表单来订购整个寄存器的更新。 当一个服务器收到一个推力时，它会将收到的包文件上传到blob商店，并准备在其本地复制的寄存器中更新 ref. 为了发布推取，它新增是存储在blob商店的共享运单的条目，记录了 ref更新并指向包文件。 Git 服务器随后使用比较和分页操作来更新运单。 这在解剖学上取代了表，或者说否定了变化。 如果有另一个作者在此期间更新了它，服务器会读取中间的条目，检查其参考更新是否仍然有效，并使用它已经上传的包文件进行再试。
作家可以平行地上传他们的包文件，但甚至推向无关的分行都需要更新相同的运算。 成功比较和扫描的往返时间限制了寄存器能公布新清单的频率。 低纬度的blob商店使这个步骤更快。 Batching让一位显示员同时更新了几位参考员，其代价是排队更新并增加延迟。
blob商店可以平行接收包文件，但共享清单的更新仍然相互竞争。 仅仅提高blob商店的能力并不能解决这个瓶颈。 每个复制品仍然需要从blob商店读取和索引包，才能在当地服务。 如果来信的进度超过工作进度，复制品就会落后，客户端会等待更长的时间来获取新提出的承诺。
带来我们去潘多 Pando是服务于Git寄存器的一款新架构，目标是同时向许多客户端推推量规模。 以下是如何运作：
在开发潘多之前，我们有一个预感，我们可以满足我们的要求，而不损害基特，但我们确实需要妥协一些大多数基特系统假定是要求的东西：
- 整个存储库的一致性：提供一致的、即时的快照需要整个存储库的协调。 在一个命令所有调试更新的系统中，推向无关的分支在单一的协调中心上进行争吵。
– 每个客户端独有的包文件：如果两个具有不同本地历史的客户端获取相同的分支，服务器可能需要为每个客户端生成不同的包来避免发送它们已经拥有的对象。 创建这些响应会花费CPU和磁盘 I/O，从而难以被放大到高卷阅读器。
取而代之的是，Pando为缩放写作提供了宽松的一致性模型(causal conference)，再用包文件来缩放读取。
当客户端推向分店时，潘多将所接入的包文件作为不可改变的blob存储在blob商店中。 分行每次移动时，潘多都会记录一个包含新提示的Ref版本，提供该提示的包，以及包在元数据存储所依赖的任何更早的ref版本。 blob商店和元数据商店都横向可扩展。
从 Pando 获取承诺或引用需要重放一组 ref 版本( packfiles)给客户端，重建创建 ref 版本的客户端状态。 这为整个存储库提供了一种较弱的一致性形式，同时为单个参考文献提供可序列性。 客户仅在同时向同一裁判推进时有冲突。 一个读取几个分支的客户端不能假设这些读取描述整个寄存器的单个时间点。
这是因果关系的一种形式。 为了取出参考文献，读者观察出出其作者的历史。 在一推中，Pando控制客户端和服务器所商定的共享历史，确保客户端生产更方便共享的包文件。
因此，Pando并不提供Git操作期间所有引用的时点状态，也不命令在克隆或取取回中校对更新。 具体来说，假设树枝和树枝都更新了
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Scaling Git at OpenAI
Building agentic-scale scale source control while investing in the Git ecosystem.
Over the past year, the size and demands on our monorepo have increased at a staggering pace. It was clear that we needed to revamp our source control system to scale to the level of concurrent work that Codex makes possible.
We built Pando, a new backend for Git repositories, without interrupting any engineering work or changing how our team uses Git. Pando now powers all development in OpenAI’s monorepo. Pando is designed to meet two ambitious goals: first, horizontally scalable writes, optimized for concurrent pushes to many branches. And second, maximize read throughput, so clients can download data fast enough to fully saturate their available network bandwidth.
We met these goals while retaining compatibility with the Git protocol and client, enabling both our engineers and agents to seamlessly adopt Pando into their existing workflows. In production, Pando cut median push latency by nearly 5x. It now delivers nearly a petabyte of Git data per day.
Here, we’ll cover our motivations for building a new source-control system, along with why we chose to retain compatibility with Git. We’ll compare our system with common approaches to scaling Git, describing why Pando is uniquely capable of meeting OpenAI’s scalability needs. We also discuss what’s next, and how we’ll continue contributing to Git to support agent-scale work.
Codex has changed the scale and pace of development at OpenAI, enabling engineers to do far more work in parallel. That means more branches and pushes, more merges, and more fetches and clones from development environments, research jobs, and CI.
These demands compound. One engineer now has several changes ready for CI at once, and each push can set several jobs fetching the same commit.
Starting around the end of last year, the systems supporting our monorepo were struggling to keep up. We suffered slow fetches and pushes along with failed merges, with reliability at times falling below 90%. Had we reached a limit of Git itself, or of the way we served our monorepo? Git grew out of a community that exchanged patches to the kernel via email. For more than two decades, developers have adapted their tools around Git as the way they build software has changed. When we started to feel friction in our own monorepo, it mattered to us that we could keep building on that work.
We saw a way to keep that familiar workflow while rebuilding the backend to handle pushes to different branches in parallel and serve the same history to many readers concurrently. Building on Git as the foundation of our source-control workflow meant that improvements to the client could be useful beyond OpenAI. We wanted to contribute those improvements from our copy of Git(opens in a new window) to upstream, sharing them with the broader community.
Scaling a source control system has two essential dimensions: many writers pushing to different branches, and many readers fetching the same history. Pando optimizes for both.
First, Pando coordinates updates to each branch independently. Updates to unrelated branches can proceed in parallel, and a reader fetching a branch gets the history needed for that revision. The tradeoff is that a reader looking at several branches can see them at different points of time.
Second, Pando reuses packfiles (Git’s format for compressing multiple objects together) across fetches instead of generating an individual packfile for each client. A client may download some objects it already has, but the server can serve the same bytes to many clients with little computation.
These choices let us scale writes across branches and serve reads from caches. To see how they fit together, let’s start with how Git stores and transfers history.
Git must be able to keep two parts of a repository in sync: objects and references.
A Git repository records snapshots of files and the history connecting them as a graph of objects. An object may be either a blob, tree, commit, or tag. Blobs hold the content of an individual file, and trees associate names with blobs or other trees to form a directory structure. Commits point to a root tree and zero or more parent commits, forming the history of your repository.
Objects are identified by a hash of their type and contents. Objects are also immutable: recording a changed file creates a new blob, while unchanged files and directories can reuse existing objects.
Branches and tags are references, or “refs”, that point to commits. Committing on a branch creates a new commit and updates the branch’s ref to point to it.
Git stores objects in either a “loose” or “packed” state. Loose objects are stored in individual files identified by their object ID. Packed objects are stored together in a packfile, enabling similar objects to be stored as deltas against one another.
In Git, an object is reachable from a commit if Git can find it by following links from that commit through its parents, trees, and blobs. During a fetch, the client tells the server which commits it wants and which it already has. The server finds the objects reachable from the wanted commits, excludes those it knows the client already has, and sends the remainder as a packfile in the same format Git uses on disk. A full clone starts without any objects, so it needs all the objects reachable from the history being cloned.
A push transfers objects in the other direction by asking the server to update one or more refs. The client uses the server’s set of advertised refs to avoid sending objects the server already has. The pack being sent can therefore depend on existing objects: receiving the pack alone is not enough to reconstruct the new history.
Replicating a Git repository means keeping both objects and references available across multiple servers. Before a replica can serve a reference’s new value, it must be able to supply the objects reachable from that value. The service also needs to coordinate competing reference updates, i.e., pushes that try to move the same branch from the same old tip to different new commits. Systems that impose a single order on all writes to a repository can make pushes to unrelated branches wait on one another.
While designing Pando, we evaluated conventional approaches for scaling source control using Git: one that coordinates ref updates across repository replicas using three-phase commit (3PC, a protocol for coordinating a commit across multiple nodes), and one that orders updates through a shared write-ahead log (WAL, a record of changes written before they’re applied) in blob storage.
In the three-phase commit design we modeled, the Git service keeps a complete copy of the repository on several replicas. When any server receives a push, the incoming packfile is copied and sent to all replicas in parallel. To publish the push, a majority of replicas must accept the ref update. The server managing this is the coordinator, and it requires multiple rounds of communication between the coordinator and each other replica.
To accept a reference update, each replica locks its copy of the ref and checks that it still points to the commit the client expects. The coordinator then sends a pre-commit message to the replicas that accept. Once a majority acknowledges the pre-commit message, the coordinator tells them to perform the update. Any up-to-date replica can then advertise the new branch tip from its local copy of the repository.
Keeping complete replicas makes it possible to serve reads locally, and guarantees that clients get a consistent view of all references no matter which replica they pick. As a consequence, references are updated one at a time, and each push requires multiple round trips to coordinate the ref update among replicas.
Waiting for a quorum puts replica disk performance and communication latency on the push path. As write volume grows, the replicas must keep up with incoming packs as well as read traffic.
In the WAL-based design we modeled, the Git service uses a shared manifest in blob storage to order updates across the repository. When a server receives a push, it uploads the incoming packfile to a blob store and prepares the ref update in its local copy of the repository. To publish the push, it adds an entry to a shared manifest stored in the blob store that records the ref update and points to the packfile. The Git server then updates the manifest using a compare-and-swap operation. This atomically replaces the manifest, or otherwise rejects the change. If another writer has updated it in the meantime, the server reads the intervening entries, checks that its ref update is still valid, and retries using the packfile it already uploaded.
Writers can upload their packfiles in parallel, but even pushes to unrelated branches require updating the same manifest. The round-trip time for a successful compare-and-swap limits how often the repository can publish a new manifest. A low-latency blob store makes this step fast. Batching lets one manifest update several refs at the same time, at the cost of queueing updates and increasing latency.
The blob store can receive packfiles in parallel, but updates to the shared manifest still compete with one another. Increasing the blob store’s capacity alone doesn’t resolve that bottleneck. Each replica still needs to read and index packs from the blob store before it can serve them locally. If incoming writes outpace that work, replicas fall behind and clients wait longer to fetch newly pushed commits.
That brings us to Pando. Pando is a new architecture for serving Git repositories with the goal of scaling to many clients pushing simultaneously. Here’s how it works:
Prior to developing Pando, we had a hunch that we could meet our requirements without compromising on Git, but we did need to compromise on some of the things that most Git systems assume to be requirements:
- Repository-wide consistency: Providing a consistent, point-in-time snapshot of all refs requires coordination across the repository. In a system that orders all ref updates together, pushes to unrelated branches contend on a single coordination point.
- Unique packfiles for every client: If two clients with different local histories fetch the same branch, the server may need to generate a different pack for each to avoid sending objects they already have. Creating these responses costs CPU and disk I/O, making it difficult to scale to high volumes of readers.
Instead, Pando provides a relaxed consistency model (causal consistency) to scale writes, and reuses packfiles to scale reads.
When a client pushes to a branch, Pando stores the incoming packfile as an immutable blob in a blob store. Each time the branch moves, Pando records a ref version containing the new tip, the pack that supplied it, and any earlier ref versions the pack depends on in a metadata store. Both the blob store and metadata store are horizontally scalable.
Fetching a commit or reference from Pando requires replaying a set of ref versions (packfiles) to the client, reconstructing the state of the client that created the ref version being fetched. This provides a weaker form of consistency across the entire repository, while providing serializability for updates to individual refs. Clients only conflict when pushing to the same ref simultaneously. A client reading several branches cannot assume those reads describe a single point in time across the repository.
This is a form of causal consistency. In order to fetch a reference, the reader observes the history of the writer that produced it. During a push, Pando controls what shared history the client and server agree on, ensuring clients produce more easily shared packfiles.
As a consequence, Pando doesn’t provide a point-in-time state of all references during Git operations or an ordering of ref updates within a clone or fetch. Concretely, suppose that branches foo and bar are both updated
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.