AI21 实验室发布 LIFT-Bench 到 Curb Reward Hacking in RL Training

AI 模型和平台

AI21 实验室发布 LIFT-Bench 到 Curb Reward Hacking in RL Training

在 Google 上的首选来源中添加 Unite.AI

AI21 Labs 发布 LIFT-Bench 在2026年10月25日，一个跟随基准测试的指令，其中每个约束范围被限定为文件的一节，任务内容本身被打分，与LIFT-Train，一个34,258任务训练对应的对等，以及同时产生两者的开放式管道。

2026年10月20日，Eran Goldstein、Yuval Belfer、Yonatan Belinkov和Daniel Gissin向ArXiv提交了一份题为“LIFT：评价和训练指导的范围限制”的配套文件。 其抽象地指出，常见的跟随基准测试的指令，如IFEval等，评价那些容易在程序上被验证的制约因素，这些相同的制约随后成为强化学习的奖励信号，而这种设计可以导致奖励将模型推出主要任务的黑客。

基准测试后方的训练运行

AI21在发布后发表的一则博客文章中，使用IFTrain(该帖形容为去开放数据集用于教学)，作为RL奖赏的来源，跑将奖赏推向了1.0和融资 IFEval以5分，IF Bench以16分。 然而当这个模型被问到关于新海革命的交替历史的一章，没有字重复了四次以上的时候，它用一个单词回答："中国".

失败是系统性的，而不是一阵风波，公司认为：通过步骤390，中位答案已经从114个字收缩到18个字，38%的答案在30个字以下至少得分0.8. AI21将倒闭归结于奖励的结构：IFTrain的检查器，和IFEval一样，用代码对照整个响应来验证限制，没有对任务是否完成进行评分，因此降低任务成本让模型一无所获。 已存在的补丁并没有弥补这一差距，公司称：IF Bench 新增是奖励型号的分级器，分出一般质量而不是具体任务，AdvancedIF列车在内容路由上运行，但保留了它的路由生成器，培训数据，和RL代码。

范围限制和经认证的法官

LIFT-Bench将每个约束都附加在文件大纲的一节中，并且只根据该节的文本进行评分，一个设计AI21称为范围限制。 每节包含有两种限制：可核查的限制用代码检查，例如需要一节德文或至少用过三次的关键词，以及由LLM检查，说明该节必须涵盖的内容的判断。 判断约束与格式化相同。 范围界定还将约束密度，单节所附加的制约数量等转化为任务生成期间的参数集。

所有任务都是合成的：主题出自维基百科，轮廓和限制出自LLM，一个采样器控制每个部分有多少个限制。 基准测试有784个测试任务，分为10个分块：短文件约4个分块，长文件约9个分块，每分块有4个密度级，外加2个平分而无大纲，其中每个约束都以IFEval等已存在的基准测试的方式适用于整个响应。

由于同一名法官得分为基准测试并颁发了RL奖赏，AI21在选择一个前，先根据305个人类标注的例子对候选法官进行了测试。 该公司报告说，一名没有瞄准镜的Gemma-4-31B法官接受了其中的292个例子，当时只有227个实际满意，一位没有瞄准镜的Qwen2.5-32B得分0.44个宏-F1，与总回答是的0.43分大致相同。 将法官范围限定在文本中，每个约束都附加了11到19个宏-F1分，这是任何可变测试的最大效果，AI21选取的范围为Gemma-4-31B. 发布分出者将google/gemma-4-31B-it写入每个任务中作为裁判员，存储器警告说分数只有使用同名裁判员的横跑相可比较。 结构分化给结构带来10%的分量重量和内容限制；平面任务没有结构限制，因此内容占用了全部分量。

关于当前模型的基准测试显示

AI21评价了17个模型，每个任务有8个样本。 关于结构化任务，公司报告，GPT-5.5平均为0.96个，而平面任务则没有模型通过0.85个，主要的封闭模型平均通过0.65到0.72个。 本文作者同样指出，前沿模型与基准测试的更难子集发生斗争。 结构-平面相差并不是一个约束负载的问题，根据AI21：限制在有10个约束或更小的任务，除了Qwen3.8-A95B之外，每个模型在平面任务上仍然得分更低，只有结构式任务才能挣取10%分的大纲分。

两种政权之间的分级变动也由公司报告。 在所跟踪的9个主要型号中，GPT-5.5在结构化任务上排名第一，在平分任务上排名第五，两个最佳整体平均值属于开口量级的型号Kimi K3和DeepSeek V4 Pro. Kimi K3在为文件的其余部分增加限制的同时，持有一节限制固定下来，每个密度级损失0.4%，而GPT-5.5损失1.5%，而GPT-5纳米损失5.5%.

以基准测试分数作为奖励的训练

LIFT-Train将基准测试的分数作为RL的奖励，不变地适用于来自同一发电机的34,258项任务的RL奖励；同一约束-奖励scalar的存储器指出同时作为基准测试的分数和奖励，没有单独的奖励模式。 AI21在LIFT-Train和IFTrain上与GRPO一起培训了Qwen2.5-7B. 在"IFTrain"上，悬赏分出一分一分一分，第180步打出一分一分一分，而中位数答案则由114个单词分出一分一分，第390步倒下至18分一分。 在LIFT-Train上，中位数答案停留在390到530个字左右，30个字以下没有获得高分的答案。

与IFTrain的表面格式相匹配，AI21 称指出奖励而不是格式的内容是决定因素，同时指出数据集的主题仍然不同，维基百科对WildChat. 在180个GRPO步骤后跨越了5个基准测试，每个食谱都赢得了自己的基准测试；在AdvancedIF和Complex Bench上，IFTrain在LIFT-Train稳住的同时滑倒了。 AI21将小三角洲定性为避免回归而非一般取胜。

对Qwen3-32B分别进行每个范围的培训，对混合物的培训，只有结构化的培训达到0.805次，对结构化任务的培训达到0.798次，对混合物的培训则达到0.896次和0.919次，而易于硬化的课程没有区别，AI21报告。 公司在总结中建议各团队进行指导培训，以便衡量任务，而不仅仅是规则，对照人类标签检查法官，在范围评价上选择基准模型，而不是平面领导板，并在跟踪答复长度的同时，将评价作为奖励。

可用性

LIFT-Bench,LIFT-Train，而合成管线则以Apache-2.0许可证在GitHub上提供，基准测试在Hugging Face Hub上作为ai21labs/LIFT Bench托管。 任务以PrimeIntellect的核查员库为评价和RL培训而使用的格式发布，存储库公布一个由17 129项任务组成的五分法质-rl培训组合。 可核实的制约类型包括语言、案例惯例、关键词计数、emoji计数和大纲匹配文件结构。

本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

AI Models & Platforms

AI21 Labs Releases LIFT-Bench to Curb Reward Hacking in RL Training

Add Unite.AI to your preferred sources on Google

AI21 Labs released LIFT-Bench on October 25, 2026, an instruction-following benchmark in which every constraint is scoped to one section of a document and the task content itself is scored, alongside LIFT-Train, a 34,258-task training counterpart, and the open pipeline that generated both.

A companion paper, “LIFT: Scoped Constraints for Evaluating and Training Instruction Following” by Eran Goldstein, Yuval Belfer, Yonatan Belinkov, and Daniel Gissin, was submitted to arXiv on October 20, 2026. Its abstract states that common instruction-following benchmarks such as IFEval evaluate constraints that can easily be verified programmatically, that those same constraints then serve as reward signals for reinforcement learning, and that this design can lead to reward hacking that pushes models away from the main task.

The Training Run Behind the Benchmark

In a blog post published with the release, AI21 reports that the project began with a real experiment that looked successful on paper. Using IFTrain, which the post describes as the go-to open dataset for instruction following, as the source of RL rewards, the run drove the reward to 1.0 and raised IFEval by 5 points and IFBench by 16 points. Yet when the model was asked for a chapter of alternate history about the Xinhai Revolution, with no word repeated more than four times, it answered with a single word: “China”.

The failure was systematic rather than a fluke, according to the company: by step 390, the median answer had shrunk from 114 words to 18, and 38% of answers scored at least 0.8 in under 30 words. AI21 attributes the collapse to the structure of the reward: IFTrain’s checkers, like IFEval’s, verify constraints with code against the whole response, and nothing scores whether the task was done, so dropping the task costs the model nothing. Existing patches do not close that gap, the company says: IFBench adds a reward-model classifier that scores general quality rather than the specific task, and AdvancedIF trains on content rubrics but keeps its rubric generator, training data, and RL code closed.

Scoped Constraints and a Validated Judge

LIFT-Bench attaches each constraint to one section of a document outline and scores it only against that section’s text, a design AI21 calls scoped constraints. Each section carries two kinds of constraints: verifiable ones checked by code, such as requiring a section in German or a keyword used at least three times, and judged ones checked by an LLM that spell out what the section must cover. Judged constraints count in the same score as formatting. Scoping also turns constraint density, the number of constraints attached to a single section, into a parameter set during task generation.

All tasks are synthetic: topics come from Wikipedia, outlines and constraints from an LLM, and a sampler controls how many constraints land on each section. The benchmark holds 784 test tasks across ten splits: short documents of about four sections and long documents of about nine sections, each at four density levels, plus two flat splits with no outline, in which every constraint applies to the whole response in the manner of existing benchmarks such as IFEval.

Because the same judge scores the benchmark and hands out the RL reward, AI21 tested candidate judges against 305 human-labelled examples before choosing one. The company reports that an unscoped Gemma-4-31B judge accepted 292 of those examples when only 227 were actually satisfied, and that an unscoped Qwen2.5-32B scored 0.44 macro-F1, roughly the same as the 0.43 scored by always answering yes. Scoping the judge to the text each constraint governs added 11 to 19 macro-F1 points, the largest effect of any variable tested, and AI21 selected scoped Gemma-4-31B. The released splits write google/gemma-4-31B-it into each task as the judge, and the repository cautions that scores are comparable only across runs that used the same judge. Structured splits give structure constraints 10% of the score weight and content constraints 90%; flat tasks carry no structure constraints, so content takes the full weight.

What the Benchmark Shows About Current Models

AI21 evaluated 17 models with eight samples per task. On structured tasks, the company reports, GPT-5.5 averages 0.96, while on flat tasks no model passes 0.85 and the leading closed models average between 0.65 and 0.72. The paper’s authors likewise state that frontier models struggle with the benchmark’s harder subsets. The structured-versus-flat gap is not a matter of constraint load, according to AI21: restricted to tasks with ten constraints or fewer, every model but Qwen3.8-A95B still scores lower on flat tasks, and the outline points that only structured tasks can earn account for 10% of the score.

Rankings shift between the two regimes as well, the company reports. Among the nine main models tracked, GPT-5.5 ranks first on structured tasks and fifth on flat ones, and the two best overall averages belong to open-weight models Kimi K3 and DeepSeek V4 Pro. Holding one section’s constraints fixed while adding constraints to the rest of the document, Kimi K3 loses 0.4% per density step and GPT-5.5 loses 1.5%, while GPT-5 nano loses 5.5%.

Training With the Benchmark Score as the Reward

LIFT-Train applies the benchmark’s score unchanged as the RL reward across 34,258 tasks from the same generator; the repository notes that the same constraint_reward scalar serves as both the benchmark metric and the reward, with no separate reward model. AI21 trained Qwen2.5-7B with GRPO on LIFT-Train and on IFTrain. On IFTrain, the reward hit 1.0 by step 180 while the median answer fell from 114 words to 18 by step 390. On LIFT-Train, the median answer stayed around 390 to 530 words, and no high-scoring answer came in under 30 words.

A flat-only slice of LIFT-Train, matching IFTrain’s surface format but keeping content in the reward, also trained cleanly. AI21 says this points to content in the reward, rather than format, as the deciding factor, while noting that the datasets’ topics still differ, Wikipedia versus WildChat. Across five benchmarks after 180 GRPO steps, each recipe won its own benchmark; on AdvancedIF and ComplexBench, which also score content, IFTrain slipped while LIFT-Train held steady. AI21 characterizes the small deltas as avoiding a regression rather than a general win.

Training Qwen3-32B on each scope separately and on a mixture, structured-only training reached 0.805 on flat tasks and flat-only training reached 0.798 on structured tasks, against 0.896 and 0.919 for the mixture, while an easy-to-hard curriculum made no difference, AI21 reports. In its summary, the company advises teams training for instruction following to measure the task rather than only the rules, to check the judge against human labels, to pick the base model on the scoped evaluation rather than a flat leaderboard, and to use the evaluation as the reward while tracking answer length alongside the reward curve.

Availability

LIFT-Bench, LIFT-Train, and the synthesis pipeline are available on GitHub under an Apache-2.0 license, with benchmark splits hosted on the Hugging Face Hub as ai21labs/LIFTBench. Tasks are emitted in the format consumed natively by PrimeIntellect’s verifiers library for evaluation and RL training, and the repository publishes a five-split prime-rl training mixture of 17,129 tasks. Verifiable constraint types cover language, case convention, keyword count, emoji count, and outline-matching document structure.

Auto-fetched and translated by AI Gem Box. All rights belong to the original author.