在去年，我们一直分享我们的工作，利用大型语言模型捕捉苍蝇上的虫子
现在，我们分享我们所看到的结果和积极影响
- 我们的论文“在Meta进行实时捕获试验”，充分详述了我们最初的部署和试验情况。
每一个代码的更改都改进了我们的平台，但也增加了推出的新复杂性。 Meta的数以亿计的代码线不断演变，在代码获得之前积极主动地抓住风险，使我们能够更早地识别出风险，更有效地作出反应，并大规模提供可靠的经验。
在去年，我们分享了运用大型语言模型(LLMs)生成软件测试的研究。 更具体地说，我们与Just-in-Tests(Jit Tests)分享了我们的工作，这是一种全新的测试方法
自我们第一次使用发布之后的五个月里，今天，Catching Jit Testes不再是研究原型；它们已经是Meta基础设施的一部分，它们帮助我们的工程师捕捉更多的虫子，负载更小，假阳性的可能性更小。
捕捉吉他试验的优势
大多数基于AI的防止代码中错误的方法使用LLMs和/或代理AI来根据以往的经验来审查代码并检测断层。 他们的工作过程与人类审查者的工作过程相同，只是更快。 捕捉JiTTTestes是一种根本不同的方法(表1)：它生成可执行测试，通过变化显示具体的行为差异推出。 虽然它需要更多的计算步骤，但它捕捉到基于LLM的或基于agent的代码审查可能没有的bug类。
我们是如何在Meta 执行捕捉的
JiTTestes是自动生成并被设计为失败的，在生产降落前在代码变化(diffs)中上浮bug. 当它们被应用到像Meta那样大的编码库时，一个挑战就是区分强渔获量(即发现真虫的真实阳性失败)和弱渔获量(即可能是假阳性测试).
归结为一个具体问题：对于每一个拟议的diff，我们如何能自动生成测试，在它们引起严重问题之前找到最严重的虫子，并且以较高的真实阳性率进行测试，而不妨碍我们工程师的速度？
为了解决这一问题，我们开发了两个协同工作流程：
- “深知”工作流程，推断代码更改的意图和风险，并将生成目标定为可能的故障点。
- 一个“dodgy diff”工作流程，将每个代码更改当作变异/出错处理，并生成测试结果，以揭露其行为差异，而不考虑意图。 它含蓄地声称，" didgy " (即 " dodgy " )。
为了过滤假阳性，我们采用了一种传统的以规则为基础的方法，加上我们称之为“LLM-as-Judge”的综观法，将几个最先进(SOTA)LLMs的判断汇总起来，所有任务都是将弱渔获量归类为可能的真或假阳性。
我们还继续进行人文审查，着眼于减少审查负荷。 我们的目标是保持人类标签的一致性，但减少工程师被带进审查弱渔获的可能性。
如何运作
整个系统的工作流程分五个阶段进行：
第1阶段：瞄准和触发
该系统在前一天提交的备用能力和目标diff上进行通宵运行，使用类似于diff Risk Score(DRS)的机制，对它进行了以往代码修改方面的培训，认为它们“危险”。 这有助于确保测试生成侧重于最有可能抑制严重回归的diffs.
第2阶段：两个 Diff- Aware 测试-工作流程
dodgy diff工作流程将每个diff当作是变种，并使用以变异导出LLM为基础的测试生成，生成能区分diff行为与母行为的测试。 这里的目标是最大限度地扩大弱渔获量，并由此找到其中的强渔获量。
同时，意向意识工作流程使用LLM通过代码更改，diff标题和摘要推断diff的意图。 LLM随后描述了“风险”——试图实施意图的方式可能产生错误。 然后，它利用这些风险描述来构建变种人——每个变种人代表一个可能为推出的bug，同时实施所推断的意图。
生成测试可以杀死每个变种人(即区分变种行为和父母行为).
然后在实际测试中进行任何失败都作为弱渔获物收获。
第3阶段：LLM-即席法官(自动评估)
一旦产生弱渔获物，系统必须确定每个渔获物是假正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正正
LLM-as-Judge 综艺节目集由LLMs组成，回顾：
- 渔获物测试码
- 测试-执行结果(失败代码、消息、执行跟踪)。
- 父母和子女之间的代码变化(diff)。
- 推断的diff意图。
第4阶段：基于规则的评估员(Rub Fake)
我们以更传统的、基于规则的方法补充LLM-as-judge，称为基于规则的假杀环境(Rub Fake)。
LLM-as-Judge拥有丰富的语义学，但其判决是非决定性的，难以核实。 相形之下，Rub Fake提供了经过核实的，决定性的判断。 Rub Fake和LLM-as-Judge合作，对弱渔获物候选人进行了总体排名，排名足够高的候选人值得进一步审议。
阶段5: " 人与人 " (工程师接触)
根据LLM-as-Judge和Rub Fake的排名，系统随后决定是否接触人类工程师。
与其说是人与人接触的方法，即人类工程师审查每一项产出，我们通过人与人接触的方法减少了摩擦，即人类工程师制定标准
Meta捕捉到JiT试验的影响
随着测试系统的建立，我们的目标是让工程师参与测试结果。
至此，Meta的589名工程师(占我们外联的64%)与Catching Jit Testers进行了接触。 这些工程师报告说"Catching JiT Testers"出现了真正可操作的发现，真实阳性率达到60%.
2026年上半年，Catching JiT Tests在diff阶段抓取了74个严重(SEV)隐私bug(英语：Catching JiTT Tests)，在他们登陆生产前发挥了关键作用，并在该时间段内将隐私SEV的总体发生率降低了20%.
为什么它很重要
代理软件开发正在帮助整个行业的工程师和开发者比以往更快地写作，评论和船码。 但更快的部署也意味着需要更快的测试。 每个在生产前被捕获的bug都是终端用户不需要处理的。 LLMs能够提供额外的安全层，能够捕捉传统测试可能错过的东西
捕捉JiTTTests是一种在技术和社会上都具有可扩展性的解决方案。 将这些测试与人与人相通的方法结合起来，有助于确保工程师很少需要读取测试代码，只需要确认所观察到的行为变化是否有意。
这种方法是概括性的：工程师们及早确定成功标准，让AI执行测试，在总体层面监测结果，并在政策层面而不是每个项目上进行干预。
读取纸张
我们的论文《在Meta的时接试验生成》在FSE 2026上发表，并在Product@scale上发表，其中详细介绍了我们在Meta基础设施中与Catching JitTests进行的初步部署和测试。
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
- Over the last year we’ve been sharing our work leveraging large language models to catch bugs on the fly — a novel approach we call Catching Just-in-Time Tests (JiTTests).
- Now, we’re sharing the results and positive impact we’ve seen with Catching JiTTests deployed into production.
- Our paper, “Just-in-Time Catching Test Generation at Meta,” fully details our initial deployment and testing with Catching JiTTests.
Every code change improves our platform, but it also introduces new complexity. With hundreds of millions of lines of code constantly evolving at Meta, traditional testing can’t keep pace. Catching risks proactively, before code lands, enables us to identify risks earlier, respond more effectively, and deliver reliable experiences at scale.
Over the last year, we’ve shared our research into applying large language models (LLMs) to generate software tests. More specifically, we’ve shared our work with Just-in-Time Tests (JiTTests), a novel testing approach where we leverage LLMs to catch bugs on the fly, even ones that traditional testing might not catch.
In the five months since we first unveiled this work, we’ve deployed our Catching JiTTests approach to determine its scalability and applicability to the wider industry. Today, Catching JiTTests are no longer a research prototype; they’re part of Meta’s infrastructure, and they’re helping our engineers catch more bugs with less load and a smaller chance of false positives.
The Advantage of Catching JiTTests
Most AI-based approaches to preventing bugs in code use LLMs and/or agentic AI to review code and detect faults based on prior experience. They work through the same review process that a human reviewer would, only faster. Catching JiTTests is a fundamentally different approach (Table 1): It generates executable tests that demonstrate a concrete behavioral difference introduced by the change. While it requires more computational steps, it catches classes of bugs that LLM- or agent-based code reviews might not.
How We Implemented Catching JiTTests at Meta
JiTTests are automatically generated and designed to fail, surfacing bugs in code changes (diffs) before they land in production. When they’re applied to a codebase as large as Meta’s, one challenge is distinguishing strong catches (i.e., true-positive failures that reveal real bugs) from weak catches (i.e., tests that may be false positives).
It boils down to a specific question: For every proposed diff, how can we automatically generate tests that find the most severe bugs before they cause severe issues, and do so with a high true-positive rate and without impeding the velocity of our engineers?
To address this, we developed two tandem workflows:
- An “intent-aware” workflow that infers the intention and risks of a code change and targets test generation toward likely failure points.
- A “dodgy diff” workflow that treats every code change as if it were a mutant/bug and generates tests to expose its behavioral differences without reasoning about intent. It implicitly asserts that a diff is buggy (hence the term “dodgy”).
To filter false positives, we employed a traditional rule-based approach combined with an ensemble approach we call “LLM-as-Judge,” which aggregates judgements from several state-of-the-art (SOTA) LLMs, all tasked with classifying weak catches as likely true or false positives.
We also maintained human review, with an eye toward reducing review load. Our goal is to maintain the consistency of human labeling but reduce the likelihood of an engineer being brought in to review a weak catch.
How It Works
The entire system workflow happens in five stages:
Stage 1: Targeting and Triggering
The system runs overnight on spare capacity and targets diffs submitted the previous day that are deemed “risky” using a mechanism similar to Diff Risk Score (DRS) and trained on previous code changes. This helps ensure test generation is focused on diffs that are most likely to contain severe regressions.
Stage 2: Two Diff-Aware Test-Generation Workflows
The dodgy diff workflow treats each diff as if it were a mutant and uses mutation-guided LLM-based test generation to produce tests that distinguish the diff’s behavior from its parent. The goal here is to maximize the pool of weak catches and with it the chances of finding a strong catch among them.
At the same time, the intent-aware workflow uses an LLM to infer the diff’s intention using the code changes, diff title, and summary. The LLM then describes the “risks”— the ways an attempt to implement the intent could produce errors. It then uses those risk descriptions to construct mutants — each representing a bug that could be introduced while implementing the inferred intent.
Tests are generated to kill each mutant (i.e., distinguish mutant behavior from parent).
Those tests are then run on the actual diff. Any that fail are harvested as weak catches.
Stage 3: LLM-as-Judge Ensemble (Automated Assessment)
Once weak catches are generated, the system must determine whether each is a false positive or actually a strong catch (true positive).
The LLM-as-Judge ensemble consists of a collection of LLMs that review:
- The weak-catch test-case code.
- The test-execution outcome (failure code, message, execution trace).
- The code changes between parent and child (the diff).
- The inferred diff intent.
Stage 4: Rule-Based Assessor (RubFake)
We complement LLM-as-Judge with a more traditional, rule-based approach called Rule-Based False-Positive Killing Environment (RubFake).
LLM-as-Judge has rich semantics, but its judgements are non-deterministic and challenging to verify. By contrast, RubFake provides verified, deterministic judgments. Working together, RubFake and LLM-as-Judge produce an overall ranking of weak-catch candidates, with those ranked sufficiently highly being worthy of further consideration.
Stage 5: Human-on-the-Loop (Engineer Reach-Out)
Based on the ranking from LLM-as-Judge and RubFake, the system then decides whether to reach out to human engineers.
Rather than a human-in-the-loop approach, where a human engineer reviews every output, we’ve reduced friction through a human-on-the-loop approach, where human engineers set criteria and review outcomes at an aggregate level rather than supervising every output.
The Impact of Catching JiTTests at Meta
With the testing system in place, our goal is to get engineers to engage with its results.
As of this writing, 589 engineers at Meta (64% of our outreach) have engaged with Catching JiTTests. These engineers have reported that Catching JiTTests surfaces genuinely actionable findings, with a 60% rate of true positives.
In the first half of 2026, Catching JiTTests played a key role in catching 74 serious (SEV) privacy bugs at the diff stage, before they landed in production and lowering the overall occurrence of privacy SEVs by 20% during that timeframe.
Why It Matters
Agentic software development is helping engineers and developers across the industry write, review, and ship code faster than ever before. But faster deployment means there’s also a need for faster testing. Every bug caught before it reaches production is one that end users never need to deal with. We’ve shown that LLMs can provide an additional safety layer capable of catching what traditional tests may miss, demonstrating that AI-generated testing can scale enough to reduce organizational risk.
Catching JiTTests is a solution that is scalable, both technologically and socially. Combining these tests with a human-on-the-loop approach helps ensure that engineers rarely need to read test code and only need to confirm whether an observed behavioral change was intended.
The approach is generalizable: Engineers define success criteria early, let AI execute the testing, monitor the outcomes at an aggregate level, and intervene at the policy level rather than per item.
Read the Paper
Our paper, “Just-in-Time Catching Test Generation at Meta,” which was presented at FSE 2026 and featured at Product@Scale, details our initial deployment and testing with Catching JiTTests in Meta’s infrastructure.
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.