我们如何发现前沿模型错位
当我们训练能力越来越强的AI模型时，我们需要可靠的方法来检测当它们的行为偏离我们的意图时。 Agents在训练，评价或部署时可能行为不当并擅自采取行动，可能造成严重后果。 为了发现这些失败——并在必要时及时干预——我们监测agents的理由、它们采取的行动以及这些行动如何影响其环境。
Hugging Face事件在监控内部编码agents已有一段时间，为了应对这一事件，以及我们模型的日益增强的能力，我们的错配监测器现在覆盖了我们跨越更广阔的地表的前沿模型，包括训练、评估、内部使用，以及视产品表面的不同而进行修改的外部部署。
为了了解这个扩展的系统如何能够帮助防止Hugging Face事件，我们用我们强化的显示器重新播放了那集中记录的行为。 在模拟中，我们的监测系统在agents首次逃出沙盒约30分钟后通知了我们的安全小组——在Hugging Face的系统在实际事件中被突破前30多小时。 我们认为，这一警告会使我们的响应者有时间在我们的模型能够获得外部服务之前就中断活动。
9月，我们看到在现场训练中，反应人员如何使用警报。 一个模型找到了一种方法，从一个本应孤立的环境到达互联网(在新窗口打开的). 我们的监视器在15分钟内标记了突破，一个响应者在3分钟后开始检视。 然后，我们在加强包含agents的系统的同时，暂停了边界强化学习。
虽然这个监测系统需要大量计算，但我们认为，随着我们的模型能力提高，模型错配的潜在后果增加，成本值得付出。 监控是我们深入防御模式行为错误所造成伤害的战略的多层之一。 校正训练旨在减少模型对有害行为的倾向；安全控制限制agent系统可以进入或影响；当这些保障措施失败时，监测帮助我们发现仍然存在的故障，在伤害发生前进行干预，并从中吸取教训。 我们认为，所有模型开发者都应该监测他们最能胜任模型的行为。
以下是我们内部监控系统如何运作、至今发现的情况
对于至少和GPT‐6 Astra一样有能力的模型，我们监测每个强化学习(RL)运行和评价，以及支持工具使用的任何操纵。 对于大多数工作量，包括RL和评价，我们还监测我们所评估的模型，认为这些模型比GPT‐5.5思维能力更高。 我们还监控所有内部的Codex流量，无论使用的型号如何。 该系统的一个版本还监测外部工具的使用情况，这些工具在我们的备灾框架下超过网络安全能力临界阈值。
为了在高效计算的同时发现事件，我们设计了监测系统，在初步筛选阶段，我们花费的计算相对较少，我们希望尽可能多地抓住与案件有关的案件：这些阶段旨在传递案件，即使证据不明确。 以后的阶段可以花更多的时间计算每个案例，以找出发生了什么，模型的行为是否相关，以及我们是否需要干预。
我们的初始层(L0)使用轻量级分类器，我们称之为活化探测器。 他们寻找模型内部激活的模式，并将标有标记的个案提交下一层审查。 我们调调这些探测器高召回，接受一些虚假的警报来降低丢失有关行为的机会。
因为我们仍在评估探测器如何检测不熟悉的错位形式，审查样本有助于我们识别探测器可能错过的行为由于其敏感性，我们有我们的前沿RL运行和内部Codex流量跳过这个层，直接去下一层。
在下一层(L1)，一个基于模型的审查员检查了agent最近的对话，思维链和工具的使用。 它寻找欺骗的迹象，未经授权的行动，以及操纵评价的企图。 审查者使用agent在行动时的同一上下文窗口这样做。 如果agent先前的工作为所审查的上下文窗口作了总结，审查者就会看到这一总结。
对于前沿的RL运行和内部的Codex流量，我们在每个转弯后端进行L1检查，每当agent总结其早先的工作，在其上下文窗口中腾出空间。 我们现在要按时间表进行，检查间隔时间不超过30分钟。
agent的行动，当L1评论员标出案件时，它被送到我们的下层(L2)，这个调查的agent可以检查评价的目的，如果有完整的笔录，就可以在agent的所有上下文上读回windows，看看它如何得出行动。 被调查的agent也可以在一个孤立的环境中写出并运行代码来检查相关的证据。 然后，它利用发现重建事件，识别重复报告，并就案件的严重性作出决定。
我们使用不同的调查agents 不同的目的。 我们的安全调查员寻找可能需要有人干预的迹象，例如当agent逃脱了它的沙盒或未经授权进入一个系统时。 同时，我们的配合调查员寻找值得研究的行为即使它没有立即的安全风险。 在使用两名调查员的情况下，每个调查员都审查我们L1层标出的所有案件。 我们目前正在扩大对内部工作量的协调统一调查的使用。
对于敏感的工作量，包括内部的Codex使用、评价和RL运行，我们下一层(L3)的进一步检查借鉴了对话之外的证据。 安全调查员将笔录与网络日志，其他系统遥测，以及以前案件的证据进行比较，以决定是否提醒人类反应者注意。 在L3中，统一审查员还比较各案件的报告，以查明严重失误和反复出现的模式，并更深入地审查前沿RL运行情况。
对于跳过这层层的工作量，L2安全调查员可以直接提醒响应者。
启动呼叫后，最优先的安全警报会到达我们安保和培训小组的成员，一般是在最初警报后15分钟之内。 答复者审查证据，审查受影响的系统，并采取行动控制事件。 我们如何跟进不太紧迫的调查结果，例如不会造成直接安全风险的调整失败，取决于相关工作量。
虚假的警报使得难以使每个工作量24小时不间断地呼救。 对于前沿RL运行，最高优先的警报可以在任何时间呼呼人类响应者。 对大多数其他交通来说，我们仅限于工作时间。 Page目前因内部的Codex交通被禁用。 为了在警报审查期间限制潜在的伤害，一些关键的警报可以自动地暂停受影响的工作量，而不等待响应者采取行动。 如果响应者没有在呼叫后30分钟内排除最高优先警报
在Hugging Face事件期间，我们隔离了内部模型原型，加密了它的重量，并限制其访问。 我们还在推出额外保障措施期间暂停了所有相关的工作量。 我们现在正探索如何使这些步骤成为我们对策的标准部分，这样我们的团队就可以在另一起严重事件中采取适当的行动。
我们的监测人员既要寻找可能需要紧急干预的安全失败，又要寻找需要调查和研究的不协调事件。 我们的监控人员在15分钟之内使用DNS(在新窗口打开)进入互联网现场
我们公布的错配报告(在新窗口打开)描述了我们监测系统所发现的其他事件。 在一个案例中，agent利用了一种测试工具来恢复(在新窗口打开)任务故意扣留的源代码。 该工具在其错误消息中披露了代码，而agent在其解决方案中使用了该代码。 在另一个版本中，agents搜索缺失的任务数据将一个共享的软件寄存器转换为跨越单独培训样本的留言板(在新窗口打开).
调查这些案件可以使我们发现弱点，以补上、隔离界限来强化，或在调整培训和评估中处理行为。 比较一下我们的监控者捕获到的不同案例，也帮助我们研究训练期间的行为变化
以下分类描述我们监测系统旨在检测的行为。
见类别定义
为了确保它继续有效，我们不断评价和改进我们的监测系统。 能够发现已知失败的监视器可能会错失不熟悉的行为，而监测系统的更新会引入新的弱点和监督。 为了确保我们的监测员能够看到和了解我们的模型正在做什么，我们以各种方式评估了它们的性能：通过记录和模拟模型错配事件，随机审核我们目前的流量，以及红色团队的努力，看看我们的agents是否能够躲避或者去功能化他们的监测员。
我们在中央仪表板上追踪每个阈值的召回情况，并测试监测系统的拟议修改如何在部署之前影响召回情况。
我们重放记录的事件测试我们的监视器在孤立环境中发生的攻击因为我们知道每个案件发生的情况，我们可以检查显示器是否识别出行为，当一个事件需要紧急反应时，它是否赋予了这个案件足够高的优先地位，以提醒一个响应者。
所记录的事件来自模式培训、评价和内部使用。 其中一些是在监测系统存在之前或通过独立调查发现的；另一些则被我们的监测员抓住，现在帮助我们检查更新情况，以保持他们发现已知问题的能力。 其中几起事件已包含在我们公开的错位披露中
我们还利用模拟攻击来测试过去事件可能无法弥补的失败。 在审调处-repo Sabotage v2(在新窗口打开)中，agent在追求隐藏目标的同时执行普通的编码任务，如禁用监测或部署未经批准的模型检查站；相关服务在评价环境内重建，这样我们就可以检查攻击是否成功而不暴露出真正的系统。 在Sneaky Sandbox Escape(在新窗口打开)中，agent
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
How we detect frontier model misalignment
As we train increasingly capable AI models, we need reliable ways to detect when their behavior departs from what we intended. Agents can misbehave and take unauthorized actions during training, evaluation, or deployment, with potentially serious consequences. In order to detect these failures—and, where necessary, to intervene in time—we monitor how agents reason, what actions they take, and how those actions affect their environment.
While we’ve been monitoring our internal coding agents for some time, the Hugging Face incident exposed a gap in our coverage: the incident began during internal cybersecurity evaluations on which we were not running our monitors for misaligned behavior. In response to that incident, and to the increasing capabilities of our models more generally, we’ve substantially expanded our system for monitoring agents’ behavior. Our misalignment monitors now cover our frontier models across a much wider variety of surfaces, including training, evaluation, internal use, and, with modifications depending on the product surface, external deployment.
In order to see how this expanded system could have helped prevent the Hugging Face incident, we replayed the behavior recorded in that episode with our strengthened monitors in place. In that simulation, our monitoring system alerted our security team about 30 minutes after the agents first escaped their sandbox—and more than 30 hours before Hugging Face’s systems were breached in the actual incident. We believe that this warning would have given our responders time to interrupt the activity well before our models were able to access external services.
In September, we saw how responders used an alert during a live training run. A model found a way to reach the internet(opens in a new window) from an environment that was meant to be isolated. Our monitors flagged the breach within 15 minutes, and a responder began reviewing it three minutes later. We then paused frontier reinforcement learning while we strengthened the systems that contain our agents.
While this monitoring system demands substantial compute, we think that cost is worth paying as our models become more capable and the potential consequences of model misalignment increase. Monitoring is one of several layers in our strategy of defense in depth against harm from misaligned model behavior. Alignment training aims to reduce models’ propensities toward harmful behavior; security controls limit the systems an agent can access or affect; and, when those safeguards fail, monitoring helps us detect the failures that remain, intervene before harm occurs, and learn from what went wrong. We believe that all model developers should monitor the behavior of their most capable models.
Below, we describe how our internal monitoring system works, what it has found so far, and how we’re evaluating and improving its performance.
For models at least as capable as GPT‑6 Astra, we monitor every reinforcement learning (RL) run and evaluation, as well as any harness that supports tool use. For most workloads, including RL and evaluations, we also monitor models we assess to be more capable than GPT‑5.5 Thinking. We also monitor all internal Codex traffic, regardless of the model being used. A version of this system also monitors external tool use for models above the Critical threshold for cybersecurity capabilities under our Preparedness Framework, as described below in this post.
In order to detect incidents while using compute efficiently, we’ve designed our monitoring system to follow a multilayer architecture. We spend relatively little compute on the first screening stages, where we want to catch as many concerning cases as we can: those stages are designed to pass cases along, even when the evidence is ambiguous. Later stages can spend more compute on each case to work out what happened, whether the model’s behavior is concerning, and whether we need to intervene.
Our initial layer (L0) uses lightweight classifiers we call activation probes. They look for concerning patterns in the model’s internal activations, and pass flagged cases to the next layer for review. We tune these probes for high recall, accepting some false alarms to reduce the chance of missing concerning behavior.
Because we’re still evaluating how well the probes detect unfamiliar forms of misalignment, we also send the next layer a random 5 percent of the traffic that our probes don’t flag. Reviewing that sample helps us identify concerning behavior the probes may have missed. Because of their sensitivity, we have our frontier RL runs and internal Codex traffic skip this layer and proceed directly to the next one.
At the next layer (L1), a model-based reviewer examines the agent’s recent conversation, chain of thought, and tool use. It looks for signs of deception, unauthorized actions, and attempts to manipulate an evaluation. The reviewer does this using the same context window the agent had when it acted. If the agent’s earlier work has been summarized for the context window under review, the reviewer sees that summary.
For frontier RL runs and internal Codex traffic, we run an L1 check at the end of each turn and whenever the agent summarizes its earlier work to make room in its context window. We are now moving to a schedule with no more than 30 minutes between these checks.
An agent’s actions can look quite different once we know what it was asked to do. When the L1 reviewer flags a case, it’s sent to our next layer (L2) for further inspection by our investigative agent. This investigative agent can check the purpose of an evaluation and, where the full transcript is available, read back across all of the agent’s context windows to see how it arrived at its actions. The investigative agent can also write and run code in an isolated environment to examine the relevant evidence. It then uses what it finds to reconstruct the incident, identify duplicate reports, and make a decision about how serious the case is.
We use different investigative agents for different purposes. Our security investigator looks for signs that someone may need to intervene—such as when an agent escaped its sandbox or accessed a system without authorization. Our alignment investigator, meanwhile, looks for behavior worth studying even when it poses no immediate security risk. Where both investigators are in use, each reviews every case flagged by our L1 layer. We are currently expanding the use of alignment investigations across our internal workloads.
For sensitive workloads, including internal Codex use, evaluations, and RL runs, a further check at our next layer (L3) draws on evidence outside the conversation. The security investigator compares the transcript with network logs, other system telemetry, and evidence from earlier cases to decide whether to alert a human responder. At L3, alignment reviewers also compare reports across cases to identify serious failures and recurring patterns, and examine frontier RL runs in greater depth.
For workloads that skip this layer, the L2 security investigator can alert responders directly.
When paging is enabled, the highest-priority security alerts reach members of our security and training teams, typically within 15 minutes of the initial alert. Responders review the evidence, examine the affected systems, and take action to contain the incident. How we follow up on less urgent findings, such as alignment failures that don’t pose immediate security risks, depends on the relevant workload.
False alarms have made it difficult to keep paging enabled around the clock for every workload. For frontier RL runs, a highest-priority alert can page a human responder at any hour. For most other traffic, we limit paging to business hours. Paging is currently disabled for internal Codex traffic. To limit potential harm while an alert is under review, some critical alerts can automatically pause the affected workload without waiting for a responder to act. We’re working toward a broader safeguard that would pause affected activity if responders have not ruled out a highest-priority alert within 30 minutes of being paged.
During the Hugging Face incident, we quarantined the internal model prototype, encrypted its weights, and restricted access to them. We also paused all relevant workloads while we introduced additional safeguards. We’re now exploring how to make these steps a standard part of our response, so that our teams can take the appropriate actions in another severe incident.
Our monitors look for both security failures that may require urgent intervention and incidents of misalignment that warrant investigation and research. Notably, our monitors were able to flag a model’s attempt to access the live internet using DNS(opens in a new window) during an RL run within 15 minutes, with a human beginning review three minutes later.
Our published misalignment reports(opens in a new window) describe other incidents caught by our monitoring system. In one case, an agent exploited a testing tool to recover(opens in a new window) source code that the task had deliberately withheld. The tool revealed the code in its error messages, and the agent used that code in its solution. In another, agents searching for missing task data turned a shared software repository into a message board(opens in a new window) across separate training samples.
Investigating these cases allows us to find vulnerabilities to patch, isolation boundaries to strengthen, or behaviors to address in our alignment training and evaluations. Comparing different cases that our monitors catch also helps us study how behavior changes during training, and which situations are most likely to elicit it.
The categories below describe the behavior that our monitoring system is meant to detect.
See category definitions
In order to make sure that it remains effective, we continually evaluate and improve our monitoring system. A monitor that is able to detect known failures might miss unfamiliar behaviors, and updates to the monitoring system can introduce new weaknesses and oversights. In order to ensure that our monitors are able to see and understand what our models are doing, we assess their performance in a variety of ways: through recorded and simulated incidents of model misalignment, random audits of our current traffic, and red-teaming efforts that see whether our agents can evade or disable their monitors.
We track recall at each threshold on a central dashboard and test how proposed changes to the monitoring system affect recall before deploying them.
We replay recorded incidents and test our monitors on attacks that agents carry out in isolated environments. Because we know what happened in each case, we can check whether the monitor recognized the behavior and, when an incident called for an urgent response, whether it assigned the case a sufficiently high priority to alert a responder.
The recorded incidents come from model training, evaluations, and internal use. Some of these were discovered before the monitoring system existed, or through independent investigations; others were caught by our monitors, and now help us check that updates preserve their ability to detect known problems. Several of these incidents have been included in our public misalignment disclosures, such as one in which an internal coding agent leaked a GitHub access token to the public internet(opens in a new window).
We also use simulated attacks to test failures that past incidents may not capture. In OAI-repo Sabotage v2(opens in a new window), an agent performs an ordinary coding task while pursuing a hidden objective, such as disabling monitoring or deploying an unapproved model checkpoint; the relevant services are reconstructed inside the evaluation environment, so we can check whether the attack succeeds without exposing real systems. In Sneaky Sandbox Escape(opens in a new window), the agent
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.