我们最常问的问题之一是，我们如何在Cerebras wafer-scale芯片上部署大型模型。 一个单一的Cerebras WSE-3T wafer拥有44GB的芯片上存储器，而万亿参数的前沿模型仅需要三字节来表示其重量。
以每秒1000-2000+令牌的方式运行多倍参数模型。
我们将使用开源模型Kimi K3
到最后，你会明白：
- 我们如何把完整的模型融入芯片记忆
- 我们如何在多个芯片上映射模型
- 我们如何处理大量同时提出的请求
用Kimi K3拆开
Kimi K3是一种开源，3T参数模型，有69个线性意向层和24个密层。
它在Bench 2.1号终端(vs. GPT-5.6 Sol's 88.8%)和浏览器Comp(vs. GPT-5.6 Sol's 90.4%)上得分惊人的88.3%. 这些结果表明前沿编码和代理能力，甚至在一些基准测试上优于领先的专利模型。 详见此处Kimi K3.
举例来说，我们使用：
- 假设平均请求大小256克
- 4位重量化(如Hugging Face执行)
- 8位 KV 缓存
# 1：每层的大小
为了计算Cerebras WSE-3T wafers的总数，我们需要支持Kimi K3，让我们首先计算每个模型层的大小。
Kimi K3有3兆个参数。 如果所有3T参数每个被存储为4位，原始参数有效载荷为：
然后将有效载荷分出93个层，从而：
#2：瓦费尔数
现在我们知道每层大约需要16GB，我们可以找出每44GB饼上有多少层相配。 答案取决于层类型：
- 敏感层需要一个额外的 KV 缓存空间，因为每个密集( 完全注意) 层会存储每个符号的密钥和值。 所以，我们把一个密集的层每个饼，留下剩下的SRAM 用于它的 KV 缓存。
- 线性观察层不需要为KV缓存增加空间。 线性注意层不为每个令牌存储密钥和值，而是为每个请求保持一个小的，固定大小的状态，无论上下文长度如何。 这为每个瓦片留下了两个线性地层(2×16GB=32GB)的空间。
因此，有了Kimi K3的69个线性地层和24个密集地层，我们得到：
这使得我们总共盘点出35+24=59个盘。
#3：把花饼连在一起
Cerebras在管道中执行模型。 在Cerebras编译器绘制了跨华费尔斯区域的模型层后，运行时间绘制了华费尔斯之间流出符号的路径。 对于超快的活性，只需要在饼花之间传递活性化，这可以通过RDMA来完成过以太网。 这大约是每跳2微秒。
需要强调的是，由于每块Cerebras wafer有44GB的芯片SRAM，每层只有16GB，所以我们通常能够直接将模型的整层装入芯片上。
这意味着在多数情况下，模型重量从来不需要移动，层层也不需要被分出来。 只有激活从一个饼到另一个饼
#4：我们如何满足一个要求
现在，我们把蜡烛连接在一起，并且模型被设定出来，我们可以把每块蜡烛看作是一个装配线上的一阶段：模型对每个阶段的权重保持不变，并且请求的中间结果会穿过层层。
例如，Wafer 1运行了模型层1–2，并将其活化到wafer 2. Wafer 2运行了3–4层，并将结果传递给Wafer 3等，直至最后的wafer产生下个符号。
#5：我们如何满足许多要求
这融资是一个常见的误解。 如果一个标志必须按顺序穿过每个饼饼，实际上，系统不是以一个批量大小运行的吗？
没有！如果我们一次向下发出一个请求，等待它完成之后再开始下一个请求，这将是对管道的低效利用。 相反，我们把整个管道都装满了不同的要求。 由于模型已经被分解为管道阶段，每个阶段可以同时处理不同的请求。
第3阶段处理请求A的代号，第2阶段处理请求B的代号，第1阶段处理请求C的代号。 在我们的例子中，每个阶段都是一层。 在生产中，Cerebras使用许多更精细的分级，还有更多的分级。
每个阶段还可以处理一个微批，一个小组的请求，而不是一个单一的请求。 因此，飞行中的请求数量大致是相相相乘以微小批量大小。
为了理解取舍，更小的微管使每个请求更快地完成，而更小的微管会增加总吞吐量。
#6：我们能满足多少请求？
分配给 KV 缓存的芯片内存量决定了同时请求的最大数量。 每一次飞行中的请求都需要自己的KV缓存在每个密集层的地盘上，因此芯片上存储器的大小为同时提出的请求设定了上限。 如上所解释，我们只需要计算出密层的KV缓存要求。
对于已经有一个密集层的44GB卷轴，我们可以用：
蜡烛存储总量 - 一个密集层的大小。 在这种情况下：
计算同时请求的 #
为了找到支持任何特定模型映射的并行请求数量，我们首先计算一个256k请求要处理多少缓存：
然后计算此配置将支持的并行请求总数，我们除以：
同时提出的请求总数是任何时间通过管道的请求总数。 与GPU批量不同，这些请求不会等待对方完成
当然，无论是货币还是批量大小，都无法确切地告诉你一个推论系统是如何高效运作的。 在评价一个推论系统的吞吐量时，更有意义的衡量标准是每单位功率(如每兆瓦每秒的活量)的总信使吞吐量，如果测量时输出速度相同。
如何超越3T参数
同一方法也延伸到了更大的模型：计算出重量和请求状态所需的内存，将模型分布在饼饼上，并把它们连接到一条管道上。 每个额外的饼饼都贡献出计算和芯片上存储器，使我们能够容纳超过任何单个芯片容量的模型。
谢谢你和我们一起学习！有兴趣在Cerebras上部署模型还是用我们的推论云来建造？
- 开始我们的免费API：云.cerebras.
- 联系我们的销售团队：销售@cerebras.ai
(原始内容存档于2017-10-21). Harley Chan. 感谢阿特哈瓦·塔尔帕德，高振伟，安吉拉·杨克，杰西卡·刘克，乔伊斯·埃尔，艾玛·克尔，和罗汉·古普塔的审查和反馈。
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
One of the most frequently asked questions we get is how we deploy large models on Cerebras wafer-scale chips. A single Cerebras WSE-3T wafer has 44 GB of on-chip memory, and trillion-parameter frontier models need terabytes for their weights alone.
In this post, we’ll explain how we split a large frontier model across multiple WSE-3T wafers, to run multi-trillion parameter models at 1000-2000+ tokens per second.
We’ll use the open source model Kimi K3, with ~3 trillion parameters, as an illustrative example.
By the end, you'll understand:
- How we fit the full model into on-chip memory
- How we map the model across multiple chips
- How we handle a high # of concurrent requests
Breaking it down with Kimi K3
Kimi K3 is an open source, 3T parameter model with 69 linear-attention layers and 24 dense layers.
It scores an impressive 88.3% on Terminal-Bench 2.1 (vs. GPT-5.6 Sol's 88.8%) and 91.2% on BrowseComp (vs. GPT-5.6 Sol's 90.4%). These results demonstrate frontier-level coding and agentic capabilities, even outperforming leading proprietary models on some benchmarks. Read more about Kimi K3 here.
For this illustrative example, we use:
- A 256k assumed average request size
- 4-bit weight quantization (like the Hugging Face implementation)
- An 8-bit KV cache
#1: Size of Each layer
To calculate the total number of Cerebras WSE-3T wafers we need to support Kimi K3, let’s first calculate the size of each model layer.
Kimi K3 has 3 trillion parameters. If all 3T parameters are stored at 4 bits each, the raw parameter payload is:
Dividing that payload by 93 layers then gives:
#2: Number of Wafers
Now that we know each layer needs about 16 GB, we can figure out how many layers fit on each 44 GB wafer. The answer depends on the layer type:
- Dense layers need additional room for a KV cache, because each dense (full-attention) layer stores the keys and values for every token in every active request. So, we place one dense layer per wafer and leave the remaining SRAM for its KV cache.
- Linear-attention layers do not require additional room for a KV cache. Instead of storing keys and values for every token, a linear-attention layer keeps a small, fixed-size state per request, regardless of context length. That leaves room for two linear layers per wafer (2 × 16 GB = 32 GB).
Therefore, with Kimi K3’s 69 linear layers and 24 dense layers, we get:
This gives us a total wafer count of 35 + 24 = 59 wafers.
#3: Connecting the wafers together
Cerebras executes models in a pipeline. After the Cerebras compiler maps the model layers across regions of wafers, the runtime maps the path for tokens to flow between wafers. For ultrafast latency, only activations need to be passed between the wafers, and this can be done via RDMA over Ethernet. This is roughly 2 microseconds per hop.
It’s important to emphasize that because each Cerebras wafer has 44 GB of on-chip SRAM and each layer is only 16 GB, we are typically able to fit entire layers of the model directly on the chip.
This means that in most cases, the model weights never have to move, and layers don’t need to be split across wafers. Only activations cross from one wafer to the next.
#4: How we serve 1 request
Now that we have our wafers connected together and the model laid out, we can think of each wafer as one stage in an assembly line: the model’s weights for each stage stay in place, and the request’s intermediate results move through the layer stages.
For example, Wafer 1 runs model layers 1–2 and sends its activations to wafer 2. Wafer 2 runs layers 3–4 and passes its results to Wafer 3, and so on until the last wafer produces the next token.
#5: How we serve many requests
This raises a common misconception. If a token has to travel through each wafer in order, doesn’t each wafer sit idle while it waits for its turn? In effect, wouldn't the system be running at a batch size of one?
No! If we only sent one request down the line at a time and waited for it to complete before starting the next request, this would be an inefficient use of the pipeline. Instead, we keep the entire pipeline full with different requests. Because the model is split into pipeline stages already, each stage can work on a different request at the same time.
While stage 3 processes a token for request A, stage 2 processes one for request B, and stage 1 one for request C. In our example, each stage is a layer. In production, Cerebras uses much finer-grained stages, and many more of them.
Each stage can also process a micro-batch, a small group of requests, rather than a single request. So the number of requests in flight is roughly the number of stages times the micro-batch size.
To understand the tradeoff, smaller micro-batches make each request even faster, while larger micro-batches increase total throughput.
#6: How many requests can we serve?
The amount of on-chip memory allocated to KV cache determines the maximum number of concurrent requests. Every request in flight needs its own KV cache on each dense-layer wafer, so the size of on-chip memory sets the ceiling on concurrent requests. As explained above, we only need to calculate KV cache requirements for dense layers.
For each 44 GB wafer that already has one dense layer, we can calculate the remaining space for the KV cache with:
Total wafer storage - size of one dense layer. In this case:
Calculating the # of Concurrent Requests
To find the number of concurrent requests supported for any particular model mapping, we first calculate how much cache a single 256k request takes up:
To then calculate the total number of concurrent requests this configuration would support, we divide:
The total number of concurrent requests represents the number of simultaneous requests moving through the pipeline at any point in time. Unlike a GPU batch, those requests don’t wait for one another to finish, they stream through the pipeline continuously.
Of course, neither concurrency nor batch size tells you exactly how efficiently an inference system performs. When evaluating the throughput of an inference system, the more meaningful metric is total token throughput per unit of power (e.g. tokens per second per megawatt), when measured at the same output generation speed.
How this scales beyond 3T parameters
The same approach extends to larger models: calculate the memory needed for weights and request state, distribute the model across wafers, and connect them into a pipeline. Each additional wafer contributes compute and on-chip memory, allowing us to accommodate models that exceed the capacity of any single chip.
Thanks for learning with us! Interested in deploying a model on Cerebras or building with our inference cloud?
- Get started with our free API: cloud.cerebras.ai
- Reach out to our sales team: sales@cerebras.ai
Designs by Halley Chang. Thanks to Atharva Talpade, Zhenwei Gao, Angela Yeung, Jessica Liu, Joyce Er, Emma Call, and Rohan Gupta for their review and feedback.
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.