Anthropic正式将平面图上报了其最初的公开出品，这意味着AI实验室已经将各种以前的私人信息放入了展示。 尽管计划书中包含了眼花一现的收入 — — 乃至更大的支出 — — 你可能会期待，但迄今为止最大的部分被交给了“风险因素 ”，这些都是可以压制Anthropic大计划的东西。 或者，在这种情况下，人性本身。
虽然在大赦国际对人类的生存威胁程度方面已经做了很多工作，但它能够造成这种破坏的具体机制基本上是不透明的。 Anthropic的94页专注在风险因素上
“充分先进的人工智能系统如果得不到开发、部署和运行，再加上适当的保障、监督和安全措施，就可能对人类构成灾难性或生存性的风险，” (这是“存在风险”一词出现三次中的第一次。 )
然而，Anthropic谨慎地将这一结果与一个特定过程联系起来：递归自改进。
RSI背后的想法非常简单。 它描述了人类将AI模型的研发交给了其他AI模型的状态。 AI已经在这里扮演了重要的角色——至少有一个实验室利用了与RSI相近的东西来加速开发一个超越其重量的模型，但点燃末日谈话的情景是人类已经完全不在循环中了。
或者，正如Anthropic存档所描述的：“风险将随着AI系统大幅度地自动化AI研发而增加，如果它们加快AI的进步，超出了我们评价和捍卫新模型的能力。”
计划书一再提出，RSI可能导致AI模型的发展快于公司的保障，公司可能甚至不会意识到它已经输掉了这场比赛，直到它太迟了。 Anthropic没有立即答复征求意见的请求。
它也得到了具体的可能发生在那之后。 “随着人工智能系统更加自主，能够在较少人监督的情况下采取相应的行动，” Projectus 称指出，“存在着这样的风险，即这种系统无论是由于与人类意图不符、客观规格错误，还是自我保护行为，都有可能在没有人类干预的情况下独立和大规模地采取有害行动。” 这里是天网领地。
说到机器人，Anthropic旗下也担心这些。 “当我们的模型直接使用物理设备时，”填报称，“错误、错位行为、对抗性操纵或滥用可能导致人身伤害、财产损失、昂贵的设备或实验材料被损坏，或释放出危险物质，而这种后果可能是立即和不可逆转的。”
你可能会认为，鉴于显然存在的利益，将认真考虑是否应该继续实行区域安全倡议，更不要说接受。 在像边境AI这样具有竞争力的行业，Anthropic在计划书的其他地方描述同样的过程
称计划书指出，“我们认为这是前沿AI最重要的结构优势之一。 ”
Anthropic后来的称未能“关闭环路”，可能导致“随着竞争对手不断利用AI进一步提高自身发展能力而逐渐落后”。 换句话说，引起所有存在关切的系统是Anthropic认为对其成功至关重要的系统。
每个AI公司都在对付同样的紧张。 当前Anthropic研究员雅各布·克克森(Jacob Coxon)因大声辞职而病毒化后，他把遏制RSI放在了行业愿望榜的首位。 Google DeepMind和其他地方的雇员也有类似的启示。 国会中有多项法案，包括由伯尼·桑德斯(Bernie Sanders)共同提出的一项法案，提议禁止RSI. 同时，这些是营利性公司，有上万亿美元的风险。
导航是一件棘手的事情。 逆行自修会变成一出跑的火车来毁灭人类。 但在那之前看看它能跑多快
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Anthropic has officially filed the prospectus for its initial public offering, which means the AI lab has put all kinds of formerly private information on display. While the prospectus contains the eye-popping revenue—and even larger expenditures—you might expect, the biggest section of the filing by far is given over to “risk factors,” which are all the things that could squash Anthropic’s big plans. Or, in this case, humanity itself.
While much has been made about the extent to which AI represents an existential threat to humankind, the specific mechanisms by which it could bring about that destruction are largely opaque. If nothing else, the 94 pages that Anthropic’s prospectus devotes to risk factors make clear both where it sees the danger coming from, and the ways in which its researchers are running toward it in order to win the AI arms race.
“Sufficiently advanced AI systems could, if not developed, deployed, and operated with appropriate safeguards, oversight and security measures, pose catastrophic or existential risks to humanity,” the prospectus reads, citing concerns both from critics and regulators and from inside Anthropic itself. (It’s the first of three times the phrase “existential risks” pops up.)
Anthropic is careful to tie that outcome, though, to a specific process: recursive self-improvement.
The idea behind RSI is strikingly simple. It describes a state in which humans hand over the research and development of AI models to other AI models. AI already plays a big role here—and at least one lab has used something close to RSI to speed up development on a model that punches above its weights—but the scenario that sparks doomsday talk is one in which humans are no longer in the loop at all.
Or, as the Anthropic filing puts it: “Risks will increase as AI systems substantially automate AI research and development if, for example, they accelerate AI progress beyond our ability to evaluate and safeguard new models.”
The prospectus repeatedly makes the point that RSI could lead to AI models that develop faster than the company’s safeguards, and that the company may not even realize it has lost that race until it’s too late. Anthropic did not immediately respond to a request for comment.
It also gets specific about what might happen after that. “As AI systems become more autonomous and capable of taking consequential actions with less human oversight,” the prospectus says, “there is a risk that such systems could, whether due to misalignment with human intentions, errors in objective specification, or self-preserving behaviors, take harmful actions independently and at scale without human intervention.” This is Skynet territory.
Speaking of robots, Anthropic flags concerns about those as well. “When our models direct physical equipment,” the filing says, “errors, misaligned behavior, adversarial manipulation, or misuse could result in bodily injury, property damage, damage to costly equipment or experimental materials, or the release of hazardous substances, and such consequences may be immediate and irreversible.”
You might think, given the apparently existential stakes, that serious consideration would be given to whether RSI should be pursued at all, much less embraced. In an industry as competitive as frontier AI, though, it’s become fait accompli. Here’s how Anthropic describes that same process elsewhere in the prospectus, this time in a section about the startup’s “core advantages:”
“We refer to this recursive dynamic as ‘closing the loop,’” the prospectus says, “and we believe it is one of the most significant structural advantages in frontier AI.”
Failure to “close the loop,” Anthropic later says, could result in “falling progressively further behind as competitors continuously use AI to further enhance their own development capabilities.” In other words, the same systems that are causing all that existential concerns are the ones Anthropic sees as critical to its success.
Every AI company is dealing with this same tension. When former Anthropic researcher Jacob Coxon went viral for loudly resigning, he put curbing RSI at the top of his wish list for the industry. Employees at Google DeepMind and elsewhere have had similar revelations. There are multiple bills in Congress, including one cosponsored by Bernie Sanders, that propose a ban on RSI. At the same time, these are for-profit companies, and there are trillions of dollars at stake.
It’s a tricky dynamic to navigate. Recursive self-improvement could become a runaway train that destroys humanity. But until then, just look at how fast it can go.
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.