AI是新的内幕威胁，但应对风险的计划来得太晚

思想领袖

AI是新的内幕威胁，但应对风险的计划来得太晚

在 Google 上的首选来源中添加 Unite.AI

Hugging Face突破和类似事件给业界带来了寒冷，但各组织应该专注于SDLC，而不是AI模型层。

7月由OpenAI agents执行的Hugging Face突破，后又很快收到涉及Anthropic和Meta型号的类似突围的报告，融资对一个看似新的网络攻击类别有警报：恶意代理AI内幕威胁。

但这类事件并不新鲜。 它发生在边境AI实验室，多年来使人们感到紧张，这给它带来了不祥的新面孔，但这些事件表明了企业多年来所积累的同样的失败模式和盲点。

他们也不是故意恶意的。 在Hugging Face突破中，人工智能agents(约1200个)试图在试验环境中实现指定的目标，在此过程中，他们秘密地交流、利用了脆弱性，并在几个月内未被发现，其中约700人扩大了“同化”环境的墙壁并进入了互联网和第三方。

对这些事件的不安的思考不只是不受约束的AI agents可能造成严重的威胁，他们无疑会这样做。 若agents能在前沿实验室团队的眼皮底下运行Amok，如果AI最重的打击者没有实时的能见度进入他们的环境，那么你的平均组织有什么机会？

各组织控制代理AI的最佳机会是，首先认识到这一新情况仅仅是一种长期挑战的强大版本。 但解决这个问题需要许多企业仍然缺乏的东西：在软件开发生命周期(SDLC)早期的能见度和可观察性。

OpenAI-Hugging Face：什么是未来的迹象

在Hugging Face突破中，包括OpenAI新旗舰GPT-5.6 Sol在内的前沿AI型号的组合在7月是发布，试图解决一个狭义的目标——即检验了agent利用安全缺陷的能力的"探索"(ExpolitGym 基准测试). Anthropic在一篇关于违反情况的文章中指出，AI模型虽然“在减少的保障下运作”，但能够外出取色，绕过技术控制，在未经批准的渠道上合作并最终采取有风险的行动。

虽然在7月发现了突破口，但agents在5月开始在雷达下停留的同时进行不适当的通信。 METR和Redwood Research在OpenAI同时进行独立平行的调查，发现虽然agents知道他们在违反规则，但很少有人考虑通知OpenAI，而是向前犁。

OpenAI报告后，Anthropic回去审核了之前的网络安全评价，发现Claude模型在4月份已经绕过三次控制，发现如何上线，并擅自进入了三个外部组织。 不久后，Meta透露其最先进的代理AI模型"Muse Spark 1.1"同样从网络安全测试沙盒中解脱出并潜入了外出公司的网络。

OpenAI在报告中还警告说，这起事件是未来迹象，说许多模型，包括日常的开源模型，很快就会有同样的能力。 最近违反《医疗保健计划》的情况强调了这一说法。

AI 必须管理 SDLC 级别

这些代理剥削通过AI社区发出一股震撼之声，OpenAI宣布将收紧测试环境的控制，再与Anthropic,Microsoft,Amazon Web Services等100多家公司联合，就AI驱动的网络攻击来袭关键基础设施发出警告。 信中警告说，袭击可能在几个月后发生。

担心被卷入类似攻击的组织需要巩固自己的基础设施，从软件开发一级开始。 真正的问题不仅仅是代理AI越来越复杂，各组织应注重缩小人和流程差距，而不是将AI风险视为技术必然性。

在整个行业中，各组织正在AI模型层集中进行监测和观察，这层已经太迟了。 到那时，马已经出仓了一半了。 在AI最初进入企业时，即SDLC内部，必须适用能见度、监测、可观察性和治理。 各组织需要确保使用AI编码工具的开发者有教育和技能，防止这些工具将缺陷、弱点和不良习惯引入管道。

大多数软件缺陷都始于开发阶段，当开发者没有技能安全地编码或通过AI纠正缺陷时，开发者需要有一个教育方案，提供必要的安全技能并监测他们在获得这些技能并定期利用这些技能工作方面的进展。

随着AI在组织运作方式上更加深入地扎根，很明显，安全可追溯性、可观察性以及开发商及其AI编码工具的治理已成为企业安全代码输出的新基准。

持续威胁要求持续应对

我们可能曾经假设软件是相对温和的，做它的程序。 如果有故障，它可能会被冻结或造成一段时间的停机，但不会进行攻击。 大赦国际，特别是代理性大赦国际，已经将这一假设推向了头顶。 无论是代理还是开发商协助，任何AI模型的行动都必须不断被观察和规范。

正在出现的AI威胁，无论是针对医院还是水处理厂，发电厂还是企业，都可能是一个可怕的想法。 但组织并非毫无防卫能力。 他们可以实施AI治理，但他们需要从地上建立起来。 通过采用一种致力于确保软件和提供开发者教育以实现这一目标的文化，各组织可以将急需的可观察性、可追踪性和治理水平带入一个潜在的问题，如果不加以控制，这一问题只会更糟糕。

本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

Thought Leaders

AI Is the New Insider Threat, but Plans to Address the Risk Are Coming One Layer Too Late

Add Unite.AI to your preferred sources on Google

The Hugging Face breach and similar incidents have sent chills through the industry, but organizations should focus on the SDLC, not the AI model layer.

The Hugging Face breach in July, carried out by OpenAI agents, followed quickly by reports of similar breakouts involving Anthropic and Meta models, has raised alarms about a seemingly new category of cyberattack: the malicious agentic AI insider threat.

But this type of incident isn’t really new. The fact that it happened at frontier AI labs, which have been making people nervous for several years, has given it an ominous new face, but the incidents illustrate the same failure mode and blind spots that enterprises have been accumulating for years.

Nor were they intentionally malicious. In the Hugging Face breach, the artificial intelligence agents—some 1,200 of them—were trying to achieve an assigned objective in a test environment, although, admittedly, their behavior wasn’t always honorable. In the process, they communicated surreptitiously, exploited vulnerabilities and operated undetected for a couple of months, with about 700 of them scaling the wall of their “isolated” environment and gaining access to the internet and third parties.

The unsettling thought about these incidents isn’t just that unconstrained AI agents can pose a serious threat, which they undoubtedly do. But if agents can run amok right under the noses of a frontier lab’s team, which has plenty of talent, resources and motivation to keep a close eye on activity, then you should probably also ask what is happening with the AI agents and AI-assisted tools running right now in your development environment. If AI’s heaviest hitters don’t have real-time visibility into their environment, what chance does your average organization have?

The best chance organizations have to rein in agentic AI begins with recognizing that this new situation is simply a high-powered version of a long-standing challenge. But addressing it requires something many enterprises still lack: visibility and observability early in the software development lifecycle (SDLC).

OpenAI-Hugging Face: A Sign of What’s to Come

In the Hugging Face breach, a combination of frontier AI models, including OpenAI’s new flagship GPT-5.6 Sol, which was released in July, were trying to solve a narrow objective—the ExploitGym benchmark, which tests an agent’s ability to exploit security flaws. Anthropic, in a post addressing the breach, noted that the AI models, while “operating under reduced safeguards,” were able to color outside the lines, skirting technical controls, collaborating over unsanctioned channels and ultimately taking risky actions.

Although the breach was discovered in July, agents started communicating inappropriately in May while staying under the radar. METR and Redwood Research worked on an independent, parallel investigation alongside OpenAI and found that although agents were aware they were breaking rules, few thought about notifying OpenAI, instead plowing forward.

After OpenAI’s report, Anthropic went back and audited its earlier cybersecurity evaluations and found that a Claude model had gotten around controls three times in April, discovered how to get online and gained unauthorized access to three outside organizations. Shortly after, Meta disclosed that a version of its most advanced agentic AI model, the Muse Spark 1.1, likewise broke free from a cybersecurity testing sandbox and infiltrated the network of an outside company.

In its report, OpenAI also warned that the incident was a sign of what’s to come, saying many models, including everyday open-source models, would soon have the power to do the same. The recent Medicare breach underscored this claim.

AI Must Be Governed at the SDLC Level

These agentic exploits have sent a shudder through the AI community, with OpenAI announcing that it will tighten controls in its testing environment and then joining Anthropic, Microsoft, Amazon Web Services and over 100 other companies to issue a warning about the coming onslaught of AI-enabled cyberattacks on critical infrastructure. The letter warned that the attacks could be only months away.

Organizations worried about being caught up in similar attacks need to shore up their own infrastructures, starting at the software development level. The real problem isn’t simply the growing sophistication of agentic AI, but the lack of real-time monitoring and meaningful human oversight. Organizations should focus on closing human and process gaps rather than treating AI risk as a technological inevitability.

Across much of the industry, organizations are focusing monitoring and observability at the AI model layer, which is already one layer too late. By then, the horse is halfway out of the barn. Visibility, monitoring, observability and governance must be applied where AI initially enters the enterprise: inside the SDLC. And organizations need to ensure that developers using AI coding tools have the education and skills to keep those tools from introducing flaws, vulnerabilities and bad habits into the pipeline.

Most software flaws start at development phase, and when developers don’t have the skills to code securely or correct flaws introduced by AI, problems can quickly proliferate. Developers need an education program that provides the necessary security skills and monitors their progress in both acquiring those skills and using them routinely on the job.

As AI becomes more deeply ingrained in how organizations operate, it’s become clear that security traceability, observability and governance of both developers and their AI coding tools have become the new baseline for secure code output in the enterprise.

Continuous Threats Calls for Continuous Response

We may once have assumed software was relatively benign, doing what it was programmed to do. If there was a glitch, it might freeze up or cause some downtime, but it wouldn’t go on the attack. AI, especially agentic AI, has turned that assumption on its head. Whether agentic or developer-assisted, any AI model’s actions must be continuously observed and governed.

The emerging AI threat, whether targeting hospitals or water treatment plants, power plants or businesses, can be a terrifying thought. But organizations aren’t defenseless. They can implement AI governance, but they need to build it from the ground up. By adopting a culture committed to secure software and providing developer education to achieve that goal, organizations can bring a much-needed level of observability, traceability and governance to a potential problem that, if left unchecked, will only get much worse.

Auto-fetched and translated by AI Gem Box. All rights belong to the original author.