GLM-5.3工人AI模型中的交叉请求 KV缓存曝光(155个答复)

GLM-5.3工人AI模型中的交叉请求 KV缓存曝光，以及我们与SGLang的补救

2026年9月8日，来自克诺斯季克的安检人员加迪·埃夫龙(Gadi Evron)提醒我们，一个租户隔离问题影响到了GLM-5.3工人AI模型(@cf/zai-org/glm-5.3)，从2026年8月27日开始。 我们已经纠正了这个问题，并于9月9日恢复了模型。 我们确认，在工人AI上托管的其他模式没有受到影响。

8月27日至9月9日期间，少数使用GLM 5.3工人AI模型的客户收到了包含来自其他客户在他们之前不久向同一种模型提出的要求的数据的答复。 总共155个答复受到影响，导致超过681 280个信使。

在排除了我们自己的测试和研究者的要求后，我们找出的要求在时间和体积上都与普通的长文使用一致，而不是与有人反复地利用虫子。 我们没有发现任何蓄意剥削的证据。

今天，我们正在发表这份报告，同时发表来自SGLang的一篇文章，即我们使用的开源推论引擎作为协调披露。 这篇文章描述了引起这个问题的缺陷，我们的反应时间表，以及随后我们对生产SGLang的部署进行的更硬化。 我们还详细说明，我们继续与SGLang的上游维护者一起努力解决这一问题和类似的问题。

发生了什么事

GLM是来自Z.ai的文本生成模型，为代理编码工作流程所建；它是通过Workers AI提供的许多模型之一。 Gadi Evron于9月8日通过我们的HackerOne程序提交了一份报告，Cloudflare于次日早上转载了这期节目，并在宣布出事后40分钟内将模型下线。 然后我们部署了一个修正的配置和一个补丁的推论引擎，并于同一天恢复服务。

关于交叉请求的KV缓存暴露的具体报告是两个重叠问题的结果。 第一个是开源推论引擎SGLang的错配置，由Cloudflare使用。 我们部署了GLM 5.3，有一个1,048,576个上下文窗口，但是我们预填GPU上的KV缓存池只有681,280个令牌的空间。 在发布之前，我们意识到这种不匹配。 我们期望引擎返回任何它无法处理的即时错误，我们的长文本测试返回了成功的答复，因此我们认为这不是一个问题。

第二是SGLang如何在分类的预填/解码模式下处理产生的超长请求的出错。 一个不符合池的快取被内部标记为拒绝并被取而代之的一键占位符，但是没有中止请求并返回客户端出错，而是预填生成了占位符的 KV 缓存，并被转移到解码节点。 解码节点收到"传输完整"的消息，并假定预先填充会从先前的请求中超额刷取出KV缓存，同时对此请求有完全的快取，而事实上它使之前状态的大部分都完好无损。 解码节点随后生成了响应，同时在浏览了仍然保存来自先前请求的数据的已停用缓存页面。 因此，对超过681 280个令牌的提示的响应可能包含来自其他用户GLM 5.3会的数据。

立即补救

在使模式失效后，我们采取了以下行动来纠正这种脆弱性，然后恢复公众使用这一模式的机会：

为了修复我们的配置，我们让游泳池保持了完整的环境。 我们调整了我们的GLM 5.3配置，以匹配我们的GLM 5.2配置，它释放出足够的GPU内存，预填KV池现在持有约204万个令牌，大约是1,048 576个上下文长度的两倍。 符合上下文窗口的提示现在总是符合集合，所以拒绝路径永远无法为有效的请求所达到。 仅此就阻止了虫子被锻炼。

为了修复引擎，我们还使拒绝路径安全。 单靠池子大小是脆弱的：任意向分数化，收分数平行化，CUDA图表大小，或未来的模型版本以容易错过的方式改变池子大小。 此外，还可能存在预填调度器可以拒绝请求并触发同一种错误的其他情况。 为了防止这种情况，我们还修补了引擎，使一个载有待决拒绝的请求在分解录入队列中被退役，加上其原错误，在任何靴子夹，分配，或KV传输发生之前。 我们提出这个上游为#38704，它也让解码节点验证所传输的符号数量与它所保留的号码相符。 该修道已经于36988年的SGLang主干道上被合并为9月6日的相關故障道，但尚未被发布。 维护者们还按照我们的方法，合并了38935号，它早些时候拒绝了这些要求。

在恢复模型之前我们验证了研究者复制脚本的修改以及我们自己的金丝雀标记测试

背景：我们如何服务GLM 5.3

工人AI在分解配置中运行了带有SGLang的GLM 5.3：一组预填GPU处理快取并产生模型的KV缓存，再将缓存放在远程直接内存访问(RDMA)上，通过Mooncake，到一个生成响应的解码GPU集合。 我们之前在高性能的LLMs和较小，更快，更安全的模型的岗位上写过这个架构。

GPU内存中的KV缓存是一个共享的固定大小页面集合。 每个请求在被接受时被指定页；发布在完成时被指定页。 当前在请求之间没有将页面零化；获得页面的下一个请求会简单地覆盖它。 这种方法在推论引擎中是常见的，并且是实现最佳性能所必需的，但这意味着推论引擎负责跟踪这些内存页的所有权。 KV缓存页面所有权和寿命跟踪不仅必须处理节点内共享的所有权(KV缓存卸载和信使生成同时访问不同寿命的内存)，而且还要处理技术上与远程预填节点共享的所有权(它同时持有可变引用页面).

此外，该池的大小不是一个配置值。 SGLang在启动时从GPU内存中装入模型重量后留下的大小。 该模型所广告的上下文长度是一个单独的配置值。 没有任何东西迫使两人同意，我们的任务是检查他们是否同意。

问题一：配置错误

在我们的GLM 5.3预填节点与我们的配置，池来681,280个令牌而模型的上下文长度是1,048,576个令牌。 所选择的KV池大小由SGLang在启动时以日志形式打印出，并作为公尺被曝光。 我们意识到这种大小不匹配；然而，在发布前测试期间，我们观测到HTTP状态代码和一个在基准设定过程中没有出错框架的流体。 我们假设引擎能够优雅地处理 KV 缓存池与请求的上下文窗口之间的差额，通过流出或其他机制。

这是我们测试中的一个关键缺口：我们只在长时间内验证HTTP响应代码和引擎性能，而没有验证模型输出和一致性。 我们现在从运行的引擎直接检查KV集合大小，然后进行分门别类的部署改变，而不是从响应代码中推断出来。

第二期：SGLang 错误

在v0.5.21中，SGLang团队现在已经处理了这个bug的处理提示比KV池大小长的方式。 以下是之前发生的事情。

当超过KV游泳池的快车到达时，SGLang的调度员拒绝了。 拒绝路径(Req.set finish with bort)记录了要返回客户端的错误，并交换了单键占位符的提示，以便请求可以在不分配真实内存的情况下通过正常的批量机械排出。 在单个节点的部署上，此操作有效：占位符运行，批量完成，客户端收到错误。

在像我们运行这样的分类模式下，预填和解码的入场排队检查请求是否完成，而不是是否被拒绝。 一个被拒绝的请求自称是活的。 所以说：

- 鞋带路由器告诉解码节点的即时长度(图中为8页). 解码为请求保留了8个KV页。

- 预填录入。 如果req. finished( ) 是 False，预填录取队列会接受任何请求。 一个被拒绝的请求将其拒绝存储为完成，而非完成理由，因此完成( ) 仍然返回 False。 被拒绝的请求被接受。

- 预填前传拒绝已经用一键占位符取代了提示符。 Prefield跑出一个微不足道的前进传递一个标志。

-KV转帐 Prefill发送了一页KV来解码并发出信号，表示传输完成。

- 解码录入。 解码的入场队列有相同的完成()检查，因此接受了交接。

- 解码注意。 解码器开始生成，注意力读取所有8个保留页。 这一要求只写了P0页。 第P1页至第P7页仍然保留着它们中较早的请求：其他用户的提示和部分代。

- 回应。 客户端得到了HTTP 200而不是HTTP 400. prools tokens 匹配了长长的提示，并且回复是连贯的。 它的到来比8页的预填要快，它包含了来自其他用户的KV页面的内容。

每一次超过门槛的请求都会这样这个bug在2026年7月6日被另一个用户作为#30233号问题报告给SGLang,Cloudflare在调查这一事件时注意到了这个错误。 我们自v0.5.10起就一直向SGLang提出这类问题，这次安全事件加速了Cloudflare与SGLang团队合作已经开始的工作。 此期的补丁为：在SGLang v0.5.20中为发布，在v0.5.21和v0.5.22中为后续，并可供所有其他依赖于SGLang的操作者使用。 我们强烈推荐所有运营商将SGLang依赖性更新为v0.5.22或更晚。 具体PR的更详细情况见下文。

GLM 5.3是受此虫影响的唯一型号。 关于工人AI的所有其他分类部署在预填和解码上都有比上下文长度更大的KV集合，因此任何请求都无法到达拒绝路径。 我们把发动机固定和下面描述的硬化应用到我们为SGLang服务的每一个模型上，我们的部署检查需要它们来进行每次新的部署。

时间线

所有时间UTC.

正在加紧努力

这个bug属于Cloudflare已经工作了几个月的KV缓存存储安全bug的类。 在每一种情况下，一个页面都被解禁、重新使用或读取，而其他的网页却仍然有权利主张。 我们看到两个共同的原因。

第一个是单节点内页寿命。 解码节点上的几个任务可以同时持有同一页：令牌生成，卸载到主机内存，KV传输接收器，以及中止和出错路径。 如果其中之一在另一页仍在使用时解开页面，则可以在仍然被读取或写入时给出新的请求。 我们先前对解码卸载(#24226,#30899)和对语法错误(#30937)的双放的修补就是这种。

第二场是横跨节点的比赛。 当一个请求在预填时仍然写入其 KV 缓存以解码 RDMA 时，已经发送的写入无法被召回。 如果解码立即解码这些页面，这些页面可以被赋予另一个请求，而后期写入请求的KV缓存中。 Cloudflare通过提议的修补(#31357,#38961)和复制脚本(#32564)来为修补这场赛事做出了贡献。 在这两种情况下，结果都是被损坏的输出，或者一个用户的数据在另一个用户的反应中，没有

本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang

On September 8, 2026, we were alerted by a security researcher, Gadi Evron from Knostic, to a tenant isolation issue affecting the GLM-5.3 Workers AI model (@cf/zai-org/glm-5.3) starting on August 27, 2026. We have remediated the issue, and we restored the model on September 9. We have confirmed that no other model hosted on Workers AI was affected.

Between August 27 and September 9, a small number of customers using the GLM 5.3 Workers AI model received responses that contained data from other customers' requests made to the same model shortly before theirs. In total, 155 responses were affected, all to prompts longer than 681,280 tokens.

After excluding our own testing and the researcher's, the requests we identified are consistent in timing and volume with ordinary long-context use, not with someone repeatedly exploiting the bug. We found no evidence of deliberate exploitation.

Today we are publishing this report alongside a post from SGLang, the open-source inference engine that we use, as a coordinated disclosure. This post describes the bug that caused this issue, our response timeline, and the subsequent hardening we have conducted on our production SGLang deployments. We also detail our continued efforts with the upstream maintainers of SGLang to address this and similar bugs.

What happened

GLM is a text generation model from Z.ai that is built for agentic coding workflows; it is one of the many models available through Workers AI. Gadi Evron submitted a report on September 8 through our HackerOne program stating that very long prompts sent to @cf/zai-org/glm-5.3 on Workers AI sometimes returned text that belonged to other users' sessions. Cloudflare reproduced the issue the next morning and took the model offline within 40 minutes of declaring an incident. We then deployed a corrected configuration and a patched inference engine, and restored service the same day.

This specific report of cross-request KV cache exposure was the result of two overlapping issues. The first was a misconfiguration of the open-source inference engine, SGLang, used by Cloudflare. We deployed GLM 5.3 with a 1,048,576-token context window, but the KV cache pool on our prefill GPUs only had room for 681,280 tokens. We were aware of this mismatch before launch. We expected the engine to return an error for any prompt it could not process, and our long-context tests returned successful responses, so we did not believe this to be an issue.

The second was a bug in how SGLang handled the resulting over-length requests in disaggregated prefill/decode mode. A prompt that did not fit the pool was internally marked as rejected and replaced with a one-token placeholder, but instead of aborting the request and returning a client error, prefill generated the KV cache for the placeholder prompt and transferred it to the decode node. The decode node received a "transfer complete" message and assumed that prefill overwrote the KV cache from previous requests with the full prompt for this request, when in fact it left the majority of the previous state intact. The decode node subsequently generated a response while attending over stale cache pages that still held data from earlier requests. As a result, responses to prompts longer than 681,280 tokens could contain data from other users' GLM 5.3 sessions.

Immediate remediation

After disabling the model, we took the following actions to correct this vulnerability before restoring public access to it:

To fix our configuration, we made the pool hold the full context. We adjusted our GLM 5.3 configuration to match our GLM 5.2 configuration, which freed enough GPU memory that the prefill KV pool now holds about 2.04 million tokens, roughly twice the 1,048,576-token context length. A prompt that fits the context window now always fits the pool, so the rejection path is never reached for a valid request. This alone prevents the bug from being exercised.

To fix the engine, we also made the rejection path safe. Relying on pool size alone is fragile: any change to quantization, tensor parallelism, CUDA graph sizes, or a future model version changes the pool size in a way that's easy to miss. Furthermore, there may be other situations in which the prefill scheduler can reject a request and trigger the same bug. To prevent this, we also patched the engine so that a request carrying a pending rejection is retired at the disaggregation admission queue, with its original error, before any bootstrap, allocation, or KV transfer happens. We proposed this upstream as #38704, which also had the decode node verify that the number of tokens transferred matches the number it reserved. The fix was already on SGLang main in #36988, merged September 6 for a related failure path but not yet released. The maintainers also merged #38935, which rejects these requests earlier, following our approach.

We validated both changes against the researcher's reproduction script and our own canary-marker tests before restoring the model.

Background: how we serve GLM 5.3

Workers AI runs GLM 5.3 with SGLang in a disaggregated configuration: a pool of prefill GPUs processes the prompt and produces the model's KV cache for it, then hands that cache over remote direct memory access (RDMA), via Mooncake, to a pool of decode GPUs that generate the response. We have written about this architecture before in our posts on high-performance LLMs and smaller, faster, safer models.

The KV cache in GPU memory is a shared pool of fixed-size pages. Each request is assigned pages when it is admitted and releases them when it finishes. Pages are not currently zeroed between requests; the next request that gets a page simply overwrites it. This approach is common among inference engines and is necessary to achieve the best performance, but it means that the inference engine is responsible for tracking ownership of these memory pages. KV cache page ownership and lifetime tracking has to deal with not just shared ownership within a node (KV cache offload and token generation both access the memory with different lifetimes), but also ownership that is technically shared with the remote prefill node (which holds a simultaneous mutable reference to the page).

Furthermore, the size of this pool is not a configuration value. SGLang sizes it at startup from whatever GPU memory remains after loading the model weights. The model's advertised context length is a separate configuration value. Nothing forces the two to agree, and it is our job to check that they do.

Issue one: misconfiguration

On our GLM 5.3 prefill nodes with our configuration, the pool came to 681,280 tokens while the model's context length is 1,048,576 tokens. The selected KV pool size is printed by SGLang in logs during startup and exposed as a metric. We were aware of this size mismatch; however, during pre-launch testing we observed successful HTTP status codes and a streaming body with no error frames during benchmarking. We assumed that the engine was capable of gracefully handling the difference between the KV cache pool and requested context window via streaming or some other mechanism.

This was one critical gap in our testing: we only validated HTTP response codes and engine performance at long contexts, without validating model output and coherence. We now check KV pool size against context length directly, from the running engine, before any disaggregated deployment change ships, rather than inferring it from response codes.

Issue two: SGLang bug

In v0.5.21, the SGLang team has now addressed this bug in how it was handling prompts longer than the KV pool size. Here’s what was happening before.

When a prompt arrived that was longer than the KV pool could hold, SGLang's scheduler rejected it. The rejection path (Req.set_finish_with_abort) recorded the error to be returned to the client and swapped the prompt for a one-token placeholder so that the request could drain through the normal batch machinery without allocating real memory. On a single-node deployment this worked: the placeholder ran, the batch finished, and the client received an error.

In disaggregated mode such as we operate, the admission queues on both prefill and decode checked whether a request was finished, not whether it was rejected. A rejected request reported itself as live. So:

- Bootstrap. The router told the decode node the prompt length (8 pages in the diagram). Decode reserved 8 KV pages for the request.

- Prefill admission. The prefill admission queue admitted any request where req.finished() was False. A rejected request stored its rejection in to_finish, not finished_reason, so finished() still returned False. The rejected request was admitted.

- Prefill forward pass. Rejection had already replaced the prompt with a one-token placeholder. Prefill ran a trivial forward pass on that one token.

- KV transfer. Prefill sent one page of KV to decode and signaled that the transfer was complete.

- Decode admission. Decode's admission queue had the same finished() check, so it accepted the handoff.

- Decode attention. Decode started generating and attention read all 8 reserved pages. Only page P0 had been written by this request. Pages P1 to P7 still held what earlier requests left in them: other users' prompts and partial generations.

- Response. The client got HTTP 200 instead of HTTP 400. prompt_tokens matched the long prompt and the reply was coherent. It arrived faster than an 8-page prefill should take, and it contained content from other users' KV pages.

Every request over the threshold behaved this way. This bug had been reported to SGLang by another user as issue #30233 on July 6, 2026, which Cloudflare noticed while investigating this incident. We had been flagging these types of issues with SGLang since v0.5.10 and this security incident sped up work that Cloudflare had already started in collaboration with the SGLang team. The patch for this issue was released in SGLang v0.5.20, with follow-ups in v0.5.21 and v0.5.22, and is available for all other operators dependent on SGLang to use. We highly recommend all operators update SGLang dependencies to v0.5.22 or later. For more details on specific PRs, read on below.

GLM 5.3 was the only model affected by this bug. Every other disaggregated deployment on Workers AI has KV pools larger than its context length on both prefill and decode, so no request could reach the rejection path. We have applied the engine fix and the hardening described below to every model we serve with SGLang, and our deployment checks require them for every new deployment.

Timeline

All times UTC.

Ongoing hardening effort

This bug belongs to a class of KV cache memory-safety bugs that Cloudflare has been working to address for several months. In each case, a page is freed, reused, or read while something else still has a claim on it. We see two common causes.

The first is page lifetime within a single node. Several tasks on a decode node can hold the same page at once: token generation, offload to host memory, the KV transfer receiver, and the abort and error paths. If one of them frees the page while another is still using it, a new request can be given that page while it is still being read or written. Our earlier fixes for decode offload (#24226, #30899) and double release on grammar errors (#30937) were of this kind.

The second is a race across nodes. When a request is aborted while prefill is still writing its KV cache to decode over RDMA, the writes already sent cannot be recalled. If decode frees those pages right away, those pages can be given to another request, and the late writes land in that request's KV cache. Cloudflare has contributed to the effort to fix this race through proposed fixes (#31357, #38961) and a reproduction script (#32564). In both cases the result is corrupted output, or one user's data in another user's response, with no

Auto-fetched and translated by AI Gem Box. All rights belong to the original author.