我们能信任LLMs来判断一个研究想法是否是新颖的吗？

我们能信任LLMs来判断一个研究想法是否是新颖的吗？

今天，我们正在发布《旧思想》,《小说问题》，这是一份新的研究，研究大型语言模型如何能够可靠地评估科学研究思想的新颖性。 除了论文之外，我们正在发布一个基准测试，用于测试新颖评委的数据和代码，以及一个自动化的管道研究者可以用来建造新的评价集。

支持科学发现的不同部分。 DataVoyager帮助研究人员探索和分析结构化数据，Auto Disovery生成并测试假说，理论家将跨大型文献体的证据合成候选科学理论。 这些项目与阿斯塔一起，指出了能够帮助研究人员从理解先前的工作转向提出、测试和完善新想法的系统。

但是，产生研究思想只是使科学发现自动化的挑战的一部分。 这些想法也需要提供真正新的东西，这意味着这些系统需要一种方法来区分新的方向和大部分重新包装现有工作的方向。

这一判决常常交给另一位LLM——我们称之为新法官。 小说法官评估研究思想相对于先前的工作是否是新的，无论是单独还是与另一个思想相比。 没有标准的方法来构建一个；研究团队选择一个模型，即时，评价格式，推理预算，有时还选择一个文献检索系统。 这些选择可能对判决产生实质性影响。 然而，这些法官往往只依靠人文文件，而不是他们实际上要评价的AI产生的想法。

我们新的基准测试结果表明，新评价本身远非可靠。 LLM评委共有6名 — GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, Claude Opus 4.6 – 对评价设置的看似微小的改变对相同想法产生截然不同的判断。 在一个实验中，只要迅速改变几个字，模型在识别两个想法中哪一个更为新颖的准确性就会超过50个百分点。 检索和附加推理不能一贯地改进判断，两个目的性新颖评价系统的表现比简单的Claude Opus 4.6基线差。

如果AI要在科学发现中发挥更大的作用，我们需要更可靠的方法来判断它是否真的提出了新的东西。 我们的工作为研究这一问题提供了工具，同时表明仍有多少问题没有得到解决。

为新法官建造基准测试

为了检验法官如何可靠地将更多的新颖研究思想与不太新颖的研究思想区分开来，我们首先需要一些例子，在那些例子中，对于基础科学贡献的独创性有强烈的证据。

因此，我们围绕ICLR 2026同行评审建立了一个自动化管道。 我们不把接受纸张当作新事物的代名词，而是将审查本身作为评论的段落，审查者明确讨论了一项贡献的独创性。 这种区别是关键；只要论文在其他方面足够有力，就可以接受而不特别新颖。 同样，一份有原始贡献的被否决的文件也可能有其他弱点。

我们创建了一个由154个人类著作研究思想组成的高新颖性人才库，选择了被接受的论文在他们评分最高的10%的ICLR研究领域，我们运用了反向标准来建立低新颖性人才库：底部10%的被否决论文的145个想法，其审查者融资原创性关注，没有任何审查者正面识别出新颖性。 在对审查-采掘过程的单独检查中，一名注释员同意在50个案例中的47个案例中提取出的新颖信号-极地性；任何分歧都不会改变审查员是否被算作赞扬或质疑论文的新颖性。

这两个集合共同构成了我们的第一个评价环境——Human-Only，它考验着新颖性法官是否能够用高低新颖性的有力证据来分辨出人类作者的思想。 但在实践中，新颖的法官越来越多地被用来评价AI产生的想法。 所以我们创造了第二个设定，人类+基因。

对于这个游泳池，我们用一个简单的LLM生成器产生的154个想法来配对同样的154个经过验证的人类思想。 我们假设所产生的想法比人类创造的对等想法更不新鲜；这些是微弱的标签，并不是每个想法都经过核实的判断。 生成者只收到一个研究领域和一个提出新想法的指示，没有文献搜索、工具、迭代或反馈。 这让我们保守地比较了来自基本生成者的想法和来自ICLR上-10%的论文的意见，这些论文的审查者明确认可了其独创性。

作为补充检查，一名专家独立审查了30个人类+ 我们把人类和人工智能产生的想法重编成一个共同的结构，以检验法官是否只是对写作风格的不同作出反应。 专家在所有30对作品中都偏好人的思想，并且评断出任何一出产生的思想小说。

重要的是，我们的许多核心调查结果根本不取决于标签。 如果两个合理的评估设置为同一对想法给出了相反的答案，那么这种分歧本身就是揭示不稳定。

小改变可以产生非常不同的判决

我们利用人类著作和AI产生的研究思想集，测试了六名LLM评委——GPT-5.1,GPT-5,GPT-5,GPT-5,Claude Sonnet 4.5,Claude Opus 4.5，和Claude Opus 4.6——如何通过分点(即绝对新颖评分)和对等(即比较两个评分以找到最新颖的)比较来评价新颖性。 我们每次都有不同的因素：迅速的推理预算、法官能否看到相关文件、想法如何格式化、有多少判决被合并成最终判决、以及(对AI产生的想法来说)哪种模式产生这些判决。

表现最大的转变来自我们如何促使模型判断新颖性。

例如，我们向GPT-5.4展示了同一个人类+. 生成的一对——每对包含一个经过验证的人类作者的想法和一个AI生成的想法——带有两个提示。 一个提示告诉模型，审查人员发现每对想法中只有一个是新颖的，并要求它确定哪个；另一个只是问哪个想法是新颖的。 当我们转向更简单的即时时时，精度从93.5%下降到了40.9% — — 低于概率 — — 而模型改变了对同对一半以上的判断。

相形之下，同样的即时变化在"人与人"数据集上只转移了约7个百分点的准确度，该数据集将人类著作的理念与更强更弱的新奇证据相配。 这表明，在人类著作的研究中，新颖的法官可以显得相对稳定，但在AI产生的想法上却表现非常不同。

称，由人类提出的想法并不能幸免。 我们看到，仅人类数据集也发生了实质性变化；同样的变化使Claude Opus 4.6的精度上升了21分。 转变的幅度比AI产生的想法要小。

也没有普遍地更迅速。 例如，在个别判断想法时，放弃一项指示，警告法官不要将微不足道或增量的贡献视为新颖的提高一些法官的准确性，而降低另一些法官的准确性。 一个对一种模式行之有效的提示可能不会转移到另一种模式，换句话说。

我们还测试了一种常用的新型法官：比较两个AI生成器的想法。 我们把来自Claude Opus 4.5的构思与Claude Sonnet 4.5配对，将GPT-5.4配为GPT-5.1. 在经过测试的迅速变化中，大约四分之一到三分之一的判决在6名法官中平均有变化。 这种分歧可以在不假设哪个生成者产生出更出众的想法的情况下加以衡量，表明法官的选取即时会如何影响思想系统之间的比较。

更多搜索和推理并不一定产生更好的新颖判断。 我们给了GPT-5 6. 解决了超高推理预算和活地，迭代搜索仅限于arXiv，再与标准检索法官进行比较，每人看到5个被检索前摘要。 (这个比较中的所有法官都会在Human+Generated上进行一次评测) 代理裁判的得分为：0.78 macro-F1 – 落后于 Claude Opus 4.6分以0.89分，且与更简单的Claude Opus 4.5和GPT-5.4的设置范围大致相同 — 而按目前的API价格计算，每个决定的成本大约会增加10×. (在这方面，宏观-F1衡量法官识别新颖和非新颖思想的程度；更高是更好的。 )

尽管成本要高得多，但目的性评价人员也不更准确。 在"Human+Generated"集上，一个简单的Claude Opus 4.6即时使用低推理和多数对三通电话的投票，实现了0.854宏-F1，总评价成本为2.59美元。 相较之下，Idea Novelty Checker，一个以文献为基础的检索和再排行的管道，以30.66分的成绩得分为0.642分，而Sakana AI的AI科学家，其新颖的评价师迭代地搜索语义学者，以82.05分的成绩得分为0.772分。 (两个专职评价员都使用Claude Opus 4.6，每个决定一次呼叫).

换句话说，简单的基准比任何一种目的建造的系统都更准确和便宜。

总而言之，这些结果表明，新颖的评价对人工智能产生的想法尤其脆弱，这些系统越来越多地被要求作出判断。

二。 对小说研究进行更可靠的评价

我们的结果表明，基于LLM的新型评价仍然是一个重大的公开问题。 对人类著作研究看来可靠的法官可能会对AI产生的想法有不同的行为；帮助一个模型会伤害另一个模型的提示；而更多的搜索或计算不一定产生更一致的判决。

今后的工作应当测试法官们将实际评价的各种想法，衡量各种提示和模型的敏感性，改进以往文献的融合方式，并建设基准测试，随着科学文献的发展和AI系统的改进，可以进化。 我们的基准测试和可重复使用的数据管道是一个起点——研究人员可以把它们应用于新的同行评审、其他科学界和子孙后代的新颖法官。

自动化的想法不仅取决于提出可信的研究方向；我们还需要一些方法来说明这些想法何时真正是新的。 通过让目前新颖法官的弱点更容易衡量，我们希望这一发布能帮助研究人员建立更强大的方法来评价AI产生的研究思想。

本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

Can we trust LLMs to judge whether a research idea is novel?

Today we’re releasing Old Ideas, Novel Problems, a new study of how reliably large language models can evaluate the novelty of scientific research ideas. Alongside the paper, we’re publishing a benchmark, data, and code for testing novelty judges, plus an automated pipeline researchers can use to build new evaluation sets.

At Ai2, we’ve been developing AI systems that support different parts of scientific discovery. DataVoyager helps researchers explore and analyze structured data, AutoDiscovery generates and tests hypotheses, and Theorizer synthesizes evidence across large bodies of literature into candidate scientific theories. Together with Asta, these projects point toward systems that can help researchers move from understanding prior work to proposing, testing, and refining new ideas.

But generating research ideas is only part of the challenge of automating scientific discovery. Those ideas also need to contribute something genuinely new, which means these systems need a way to distinguish a novel direction from one that mostly repackages existing work.

That judgment is often handed to another LLM—what we call a novelty judge. Novelty judges assess whether a research idea is new relative to prior work, either on its own or in comparison with another idea. There’s no standard way to build one; research teams choose a model, prompt, evaluation format, reasoning budget, and sometimes a literature-retrieval system. Those choices can materially affect the verdict. Yet these judges are often validated only on human-written papers rather than the AI-generated ideas they are actually meant to evaluate.

Our new benchmark results suggest novelty evaluation itself is far from reliable. Across six LLM judges – GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6 – seemingly small changes to the evaluation setup produce dramatically different judgments of the same ideas. In one experiment, changing just a few words in a prompt shifts a model’s accuracy in identifying which of two ideas is more novel by more than 50 percentage points. Retrieval and additional reasoning do not consistently improve judgments, and two purpose-built novelty evaluation systems perform worse than a simple prompted Claude Opus 4.6 baseline.

If AI is going to play a larger role in scientific discovery, we need more dependable ways to tell whether it is actually proposing something new. Our work provides tools for studying that problem, while showing how much remains unresolved.

Building a benchmark for novelty judges

To test how reliably novelty judges can distinguish more novel research ideas from less novel ones, we first needed examples where there was strong evidence about the originality of the underlying scientific contribution.

So we built an automated pipeline around ICLR 2026 peer reviews. Rather than treating paper acceptance as a proxy for novelty, we mined the reviews themselves for passages where reviewers explicitly discussed a contribution’s originality. That distinction is key; a paper can be accepted without being particularly novel as long as it is strong enough in other aspects. Likewise, a rejected paper with an original contribution can have other weaknesses.

We created a high-novelty pool of 154 human-authored research ideas by selecting accepted papers in the top-rated 10% of their ICLR research area whose reviewers positively identified novelty and where no reviewer disputed it. We applied the reverse criteria to build a lower-novelty pool: 145 ideas from rejected papers in the bottom 10% whose reviewers raised originality concerns without any reviewer positively identifying novelty. In a separate check of the review-mining process, an annotator agreed with the extracted novelty-signal polarity in 47 of 50 cases; none of the disagreements would have changed whether a reviewer was counted as praising or disputing the paper’s novelty.

Together, these two pools form our first evaluation setting, Human-Only, which tests whether novelty judges can tell apart human-authored ideas with strong evidence of higher or lower novelty. But in practice, novelty judges are increasingly being used to evaluate AI-generated ideas. So we created a second setting, Human+Generated.

For this pool, we paired the same 154 validated human ideas with 154 ideas produced by a simple LLM generator. We assumed the generated ideas were less novel than their human-authored counterparts; these are weak labels, not verified judgments for every idea. The generator received only a research area and an instruction to propose a novel idea, with no literature search, tools, iteration, or feedback. This gave us a conservative comparison between ideas from a basic generator and contributions from top-10% ICLR papers whose reviewers had explicitly endorsed their originality.

As additional checks, an expert independently reviewed 30 Human+Generated pairs, heavily weighted toward cases our strongest judge got wrong, and we rewrote the human- and AI-generated ideas into a common structure to test whether judges were simply reacting to differences in writing style. The expert preferred the human idea in all 30 pairs and judged none of the generated ideas novel.

Importantly, many of our core findings don’t depend on the labels at all. If two reasonable evaluation setups give opposite answers for the same pair of ideas, that disagreement itself reveals instability.

Small changes can produce very different verdicts

Using our collections of human-authored and AI-generated research ideas, we tested how well six LLM judges – GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6 – could evaluate novelty through both pointwise (i.e., absolute novelty ratings) and pairwise (i.e., comparing two ideas to find the most novel) comparisons. We varied one factor at a time: the prompt, reasoning budget, whether the judge could see related papers, how the ideas were formatted, how many judgments were combined into a final verdict, and – for AI-generated ideas – which model generated them.

The largest shifts in performance came from how we prompted the model to judge novelty.

For example, we showed GPT-5.4 the same Human+Generated pairs – each containing one validated human-authored idea and one AI-generated idea – with two prompts. One prompt told the model that reviewers had found exactly one idea in each pair to be novel and asked it to identify which one; the other simply asked which idea was novel. When we switched to the simpler prompt, accuracy dropped from 93.5% to 40.9% – below chance – and the model changed its verdict on more than half of the same pairs.

By contrast, the same prompt change shifted accuracy by only about seven percentage points on the Human-Only dataset, which pairs human-authored ideas with stronger and weaker evidence of novelty. That suggests a novelty judge can appear relatively stable on human-authored research but behave very differently on AI-generated ideas.

That said, human-authored ideas weren’t immune. We saw substantial shifts on the Human-Only dataset as well; the same change moved Claude Opus 4.6 accuracy by 21 points. The shifts were just less severe than on the AI-generated ideas.

There was also no universally better prompt. For example, when judging ideas individually, dropping an instruction that warned judges not to treat trivial or incremental contributions as novel improved accuracy for some judges while reducing it for others. A prompt that works well for one model may not transfer to another, in other words.

We also tested a common use of novelty judges: comparing ideas from two AI generators. We paired ideas from Claude Opus 4.5 with Claude Sonnet 4.5, and GPT-5.4 with GPT-5.1. Across the tested prompt changes, roughly a quarter to a third of verdicts changed on average across six judges. This disagreement can be measured without assuming which generator produces more novel ideas, showing how the choice of judge prompt can affect comparisons between ideation systems.

More search and reasoning didn’t necessarily produce better novelty judgments. We gave GPT-5.6 Sol an extra-high reasoning budget and live, iterative search restricted to arXiv, then compared it with standard-retrieval judges that each saw five pre-retrieved abstracts. (All judges in this comparison ran once per idea on Human+Generated.) The agentic judge scored 0.78 macro-F1 – 11 points behind Claude Opus 4.6 at 0.89, and in roughly the same range as simpler Claude Opus 4.5 and GPT-5.4 setups – while costing roughly 10× more per decision at current API prices. (In this context, macro-F1 measures how well a judge identifies novel and non-novel ideas; higher is better.)

Purpose-built evaluators weren’t more accurate either, despite costing substantially more. On the Human+Generated set, a simple Claude Opus 4.6 prompt using low reasoning and majority voting across three calls achieved 0.854 macro-F1 at a total evaluation cost of $2.59. By comparison, Idea Novelty Checker, a literature-grounded retrieval and reranking pipeline, scored 0.642 at $30.66, while Sakana AI’s AI Scientist, whose novelty evaluator iteratively searches Semantic Scholar, scored 0.772 at $82.05. (Both dedicated evaluators used Claude Opus 4.6, with one call per decision.)

In other words, the simple prompted baseline was both more accurate and considerably cheaper than either purpose-built system.

Taken together, these results suggest that novelty evaluation is particularly brittle on the AI-generated ideas these systems are increasingly being asked to judge.

Toward more reliable evaluation of novel research

Our results suggest that LLM-based novelty evaluation remains a major open problem. Judges that appear reliable on human-authored research may behave differently on AI-generated ideas; prompts that help one model can hurt another; and more search or compute does not necessarily produce more consistent judgments.

Future work should test judges on the kinds of ideas they will actually evaluate, measure sensitivity across prompts and models, improve how prior literature is incorporated, and build benchmarks that can evolve as the scientific literature grows and AI systems improve. Our benchmark and reusable data pipeline are a starting point—researchers can apply them to new peer reviews, other scientific communities, and future generations of novelty judges.

Automated ideation depends on more than generating plausible research directions; we also need ways to tell when those ideas are genuinely new. By making the weaknesses of current novelty judges easier to measure, we hope this release helps researchers build more robust ways to evaluate AI-generated research ideas.

Auto-fetched and translated by AI Gem Box. All rights belong to the original author.