人工智能
修复评价偏差，在 Amazon Bedrock 上解锁更好的模型
这篇文章与Khaled Thabit、Meriem Boughrara和来自Property Finder的Mohamed Desouky共同撰写。
Property Finder的工程团队是一个数字房地产应用软件，发现其96.8%的评价标签是直接从单一模型的预测中继承的，而不是由人类专家验证的。 他们的基本真相被系统地扭曲到现行模式，使每一种选择似乎都表现不佳。 标签也有出错，但偏差是阻碍客观模型比较的原因。 当他们将视觉AI的工作量转移到Amazon Bedrock上更具成本效益的选择时，他们发现了这一点。
物业查找器每天在中东和北非处理上万个地产图像。 他们的系统每个图像评价11个质量属性。 这些包括水印检测、房间分类、家具状况、照片质量和建筑指标。 成千上万新财产上市，所有人都需要质量评估才能上线。
物业查找者需要更具成本效益的选择。 但根据这些标签来衡量，任何替代办法都处于不利地位。
最难的是找不到更好的模式。 它发现，它们的评价方法本身就是阻碍因素。
在本文中，我们分享了Property Finder如何识别评价偏差，构建无偏见的地面真理，并应用进化即时优化在Amazon Bedrock上选择正确的模型，以86%的低成本实现更高的精度。 我们走过方法和重复的例子，可以把经验教训应用于自己的工作量。
以下图说明了Property Finder和AWS Generative AI Innovation Center团队在Amazon Bedrock上用于迭代提高分类精度的即时优化循环。
图显示迭代快速优化周期。 一个属性图像由Amazon Bedrock处理，它以JSON格式输出出预测的属性。 在优化期间，这些预测与Micro F1对256图像专家附加说明的开发集进行打分。 出错分析然后检查失败的图像并识别出可视图案，这为下个即时版本提供了信息。 循环从v0跑到v27. AI助手帮助改进了提示，后期版本使用自动优化。
你的评价可能是假的
许多球队都遇到这种情况。 要比较视觉模型，所以要构建一个评价数据集。 最常用的获得标签的方法是运行您的当前模型，并将其输出作为地上真。 该型号一直在生产中运行，没有人抱怨，所以标签必须足够好。
但这创造了一个循环评估，根据它与现任者的行为相匹配程度，而不是根据实际正确性来判断任何替代办法。
Property Finder的评价数据集恰恰存在这个问题。 在标有标签的样本中，96.8%直接来自对现有视觉语言模型(VLM)的预测，而不是独立的人类验证。 任何替代模型在单一图像处理之前在结构上都处于劣势。
隐藏错误证实了其严重性。 Watermark文本提取有字符级被标记为正确的错误，包括幻觉品牌名和缺失字符。 照片分类强制将单个标签放入包含多个房间类型的图像上。
在剩余的3.2%的人类标签样本中，注释者在原规格之外有推出类，如Office出现时没有这种标签。 当答案密钥被你试图替换的模型所写入时，模型比较会失去有效性。
建立你可以信任的真相
要打破循环，你需要拥有真正领域知识的人类专家独立地做新的说明。 未提及以前的模型产出。 无法接触到现任者所预想的。
数据集范围
Property Finder的团队手持标注的356个阿联酋图像和一套单独的埃及图像。 视察队从具有代表性的行业部门未加标签的生产清单中挑选了这些产品。 本文报导了阿联酋的图片。 埃及的影像是230个影像的跨区域检查，首批256个被注解的阿联酋图像构成了用于调谐的开发集。 最终模型比较的评价集使用所有356个，在开发运行后增加了100个附加注释的图像。 一组356个图像大致足以探测出模型之间的中度差异。 个别属性内的稀有分类的例子较少，因此关于这些属性的较小差异可能无法被察觉出。
专家说明
一个策略和分析专家给所有图像注释。 他们带来了房地产上市质量标准方面的专业知识。 AWS Generative AI创新中心团队通过专门的澄清会进行系统的质量审查。 这些审查解决了模棱两可的案件，并使数据集的标签保持一致。
多标签过渡
地产图像经常通过门道显示多间房间，或者显示相重叠的质量特征。 从单一标签到多标签分类的转换缓解了人为模糊的一个主要根源。
走廊照片显示一个卧室穿过一个出入口，这是真的。 在你允许多个标签后，你测量模型得到的正确性，而不是为了合理的二次预测而惩罚它。
混合方法可以补充这种方法：来自不同家庭的模型会浮出候选的标签和分歧，人类专家会裁定每个案件，因此没有模型输出直接用作标签。 这将减轻未来数据集扩展的注释负担。
计量选择
微分F1作为主要衡量标准，因为它对每项预测的权重相等，这与总体标签正确性的业务目标相匹配。 Macro F1与它并列被报道，因为Micro F1以频繁的类为主，可以在稀有属性上隐藏弱性能。 每个标签的精确度和召回度有助于确定具体的失败模式。 每个图像的预测标签与专家的标签重叠的程度。 旋转或水印检测等9个单标签属性每个图像有一个正确的答案。 AWS团队也以精确的匹配精度，即答案与专家标签相匹配的图像份额来打分。
快速优化作为强化学习
有了不带偏见的评价，就可以开始改进。 你无法改善你无法正确测量的东西。
优化遵循类似于强化学习的循环。 将每个快取版本想做为agent，将Micro F1分数想做为奖励信号，将错误分析想做为改进的梯度。 这个想法类似于GEPA(Genetic-Pareto反射即时演化)，在这里，你突变了即时组件，评价健身能力，选择幸存者，并脚踏实地。 在此次接触中，AWS团队从错误分析中完善了大多数版本，由AI助手审查失败的图像并提议修改。
MIPROv2，一个来自DSPy库的自动即时优化框架，为v27中的相片分类任务补充了这个人工过程。
从何而起
第一个快取 (v0) 使用了通用分类说明，这里简化用于说明：
“将这个图像分为以下类别之一：生活室、厨房、卧室、浴室。.....”。
Property Finder生产基准的自有型号A，在256-图像开发集上得分为0.55微F1. 它使用了v22，一个快速调用它的模范家庭。 V21之前的分数使用更早的单标签设置
以迭代表示
AWS团队在几个候选模型上测试了早期版本，并增加了结构化的产出格式和属性特定定义。
最有启发性的修补方法来自图像级错误分析。 在早期的单一标签版本中，该模型始终将通过门道可见的房间归类为二级房间类型。 AWS Generative AI Innovation Center团队增加了基于占据了大部分帧的房间进行分类的明确指令。 后切换到多标签输出时保留了正房作为标签并允许通过门道可见的房间作为附加标签。
区域定制带来了下一个收益。 在一些地区，上市的照片带有非水印的装饰性阿拉伯文字上盖。 该案文的规则删除了一整类虚假阳性。
v21 提示将相片分类和室类切换为多标签输出，并成为每个模型的共享启动快取。 凭借它，Llama 4 Maverick 17B 来自Meta的指令在开发集上打出一分0.75的Micro F1. 从那里，AWS团队分别调制了每个型号的快取。 对于这个模型，v24目标房间大小高估，多标签处理，以及输出格式。 随后V25自动优化版本，将关键任务先行移动。 在v27中，MIPROv2调整了相片分级任务，模型的Micro F1在该任务上达到了0.76.
结束了
最后的快取，v27 PF，是 v27 加上 Property Finder 添加的规则，用于制作，带有边框或框架的图像，以及走廊。 和v27一样，它保留了阿拉伯文本规则和输出格式指令。 每个模式调用后的一个常态步骤处理所剩格式差异。
使用 v27 PF, Llama 4 马华克 17B 指令在356-图像评价集上打出一粒0.80分的Micro F1分，其中专有型号A以v22得分0.56分。 这套评价包括256个发展图像，两个模型的提示都取自这些图像。 因此，这些成绩是乐观的。
从专有型号A到Llama 4的Mavrick 17B Instruct的切换，各自在自调速器上，将开发集上Micro F1从0.55提升到0.81. 这是47%的相对收益。 单是快速调音，在Llama 4 Maverick 17B Instruct上，融资从0.75与共享的v21迅速到0.81与v27.
从失败的方法中吸取的经验教训
不是一切都成功过 2-bucket方法将水印属性和其他属性放入两个独立的API调用。 它没有带来有意义的准确性收益。 箱型图像探测也急剧下降。 没有同一提示中的水印属性，模型就误将水印覆盖为图像边框。
通用提示也失败了，不管您运行的是哪种模式。 地产图象的域名特异性太强，无法采用一刀切的指令。
结果：成本、准确性和速度
AWS Generative AI Innovation Center团队在Amazon Bedrock上以无偏见的评价框架评价了多个模型。 所有型号都从共享的v21快取起，然后被调用到开发集上的他们的模式家族。 专有型号为A得到一回调相，v22. Llama 4 马华克 17B 指令有几种，v24到v27，加上物业查找器的边缘案例规则，与专有模式A的最终比较采用了356-图像评价集。 Property Finder根据所有11个属性的精度和成本效率选择了Llama 4 Maverick 17B Instruct来完成生产工作量。 关于按区域分类的模型可用性，参见AWS区域在Amazon Bedrock中支持的模型。
下表为显示，Llama 4 Maverick 17B Instruct(v27 PF)在每个单标签属性上达到的精确匹配精度，在356-图像评价集上得分。 它在九分之六上得分90%或以上，在所有11个属性上匹配或超过专有模式A. 在相片分类上，每个相片可以有多个标签，两者得分相同。
每1 000个图像的费用
属性查找分数
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Artificial Intelligence
Fix evaluation bias to unlock better models on Amazon Bedrock
This post was co-authored with Khaled Thabit, Meriem Boughrara, and Mohamed Desouky from Property Finder.
The engineering team at Property Finder, a digital real estate application, discovered that 96.8 percent of their evaluation labels had been inherited directly from a single model’s predictions rather than verified by human experts. Their ground truth was systematically skewed toward the incumbent model, making every alternative appear to underperform. The labels also had errors, but the bias was what blocked objective model comparison. They uncovered this when migrating their vision AI workload to a more cost-effective option on Amazon Bedrock.
Property Finder processes tens of thousands of property images every day across the Middle East and North Africa (MENA). Their system evaluates 11 quality attributes per image. These cover watermark detection, room classification, furnishing status, photo quality, and construction indicators. That’s thousands of new property listings with an average of 15 images each. All need quality assessment before going live.
Property Finder needed a more cost-effective option. But measured against those labels, any alternative was at a disadvantage.
The hardest part was not finding a better model. It was discovering that their evaluation methodology itself was the blocker.
In this post, we share how Property Finder identified evaluation bias, built unbiased ground truth, and applied evolutionary prompt optimization to select the right model on Amazon Bedrock, achieving higher accuracy at 86 percent lower cost. We walk through the methodology and iteration examples, with lessons you can apply to your own workloads.
The following figure illustrates the prompt optimization loop that Property Finder and the AWS Generative AI Innovation Center team used to iteratively improve classification accuracy on Amazon Bedrock.
The diagram shows the iterative prompt optimization cycle. A property image is processed by Amazon Bedrock, which outputs the predicted attributes in JSON format. During optimization, these predictions are scored with Micro F1 against the 256-image expert-annotated development set. Error analysis then inspects failed images and identifies visual patterns, which inform the next prompt version. The loop ran from v0 to v27. AI assistants helped refine the prompts, and later versions used automated optimization.
Your evaluation might be lying
Many teams encounter this scenario. You want to compare vision models, so you build an evaluation dataset. The most common way to get labels is to run your current model and treat its outputs as ground truth. The model has been running in production and nobody complained, so the labels must be good enough.
But this creates a circular evaluation where any alternative gets judged by how closely it matches the incumbent’s behavior, not by actual correctness.
Property Finder’s evaluation dataset had exactly this problem. Of their labeled samples, 96.8 percent came directly from the predictions of the existing vision language model (VLM) rather than independent human verification. Any alternative model was structurally disadvantaged before a single image was even processed.
Hidden errors confirmed the severity. Watermark text extraction had character-level mistakes marked as correct, including hallucinated brand names and missing characters. Photo classification forced single labels onto images containing multiple room types.
In the remaining 3.2 percent of human-labeled samples, annotators had introduced categories outside the original specification, like Office appearing where no such label existed in the prompt. A model comparison loses validity when the answer key was written by the model you are trying to replace.
Building ground truth you can trust
To break the cycle, you need human experts with real domain knowledge doing fresh annotations independently. No reference to previous model outputs. No access to what the incumbent predicted.
Dataset scope
Property Finder’s team hand-annotated 356 UAE images and a separate set of Egypt images. The team selected these from unlabeled production listings across representative industry segments. This post reports results on the UAE images. The Egypt images served as a cross-regional check on 230 images and are not reported here. The first 256 UAE images to be annotated formed the development set for tuning. The evaluation set for the final model comparison uses all 356, adding 100 images annotated after the development runs. A set of 356 images is roughly sufficient to detect moderate differences between models. Rare classes within individual attributes have fewer examples, so smaller differences on those attributes might not be detectable.
Expert annotation
A strategy and analytics expert annotated all images. They brought specialized knowledge of real estate listing quality standards. The AWS Generative AI Innovation Center team conducted systematic quality reviews through dedicated clarification sessions. Those reviews settled ambiguous cases and kept the labeling consistent across the dataset.
Multi-label transition
Real estate images frequently show multiple rooms through doorways or display overlapping quality characteristics. The switch from single-label to multi-label classification alleviated a major source of artificial ambiguity.
A hallway photo showing a bedroom through a doorway is genuinely both. After you allow multiple labels, you measure what the model gets right rather than penalizing it for reasonable secondary predictions.
A hybrid approach could complement this methodology: models from different families surface candidate labels and disagreements, and human experts adjudicate every case, so no model output is used directly as a label. This would reduce the annotation burden for future dataset expansion.
Metrics selection
Micro F1 served as the primary metric because it weights each prediction equally, which matches the business goal of overall label correctness. Macro F1 was reported alongside it because Micro F1 is dominated by frequent classes and can hide weak performance on rare attributes. Per-label precision and recall helped identify specific failure modes. Per-sample intersection over union (IoU) measured how closely each image’s predicted labels overlapped the expert’s labels. The nine single-label attributes, such as rotation or watermark detection, have one right answer per image. The AWS team also scored them with exact-match accuracy, the share of images where the answer matches the expert label.
Prompt optimization as reinforcement learning
With unbiased evaluation in place, you can start improving. You cannot improve what you cannot measure correctly.
The optimization follows a loop analogous to reinforcement learning. Think of each prompt version as an agent, the Micro F1 score as the reward signal, and error analysis as the gradient for improvement. The idea resembles GEPA (Genetic-Pareto reflective prompt evolution), where you mutate prompt components, evaluate fitness, select survivors, and iterate. In this engagement, the AWS team refined most versions from error analysis, with AI assistants reviewing failed images and proposing changes.
MIPROv2, an automated prompt optimization framework from the DSPy library, complemented this manual process for the photo classification task in v27.
Where it started
The first prompt (v0) used generic classification instructions, simplified here for illustration:
“Classify this image into one of the following categories: Living Room, Kitchen, Bedroom, Bathroom…”
Proprietary model A, Property Finder’s production baseline, scored a Micro F1 of 0.55 on the 256-image development set. It used v22, a prompt tuned for its model family. Scores before v21 used the earlier single-label setup and aren’t directly comparable.
Iteration by iteration
The AWS team tested early versions on several candidate models and added structured output formatting and attribute-specific definitions.
The most instructive fix came from image-level error analysis. In the early single-label versions, the model consistently classified rooms visible through doorways as the secondary room type. The AWS Generative AI Innovation Center team added explicit instructions to classify based on the room occupying the majority of the frame. The later switch to multi-label output kept the main room as a label and allowed rooms visible through doorways as additional labels.
Regional customization produced the next gain. In some regions, listing photos carry decorative Arabic text overlays that are not watermarks. Rules for that text removed a whole category of false positives.
The v21 prompt switched photo classification and room category to multi-label output and became the shared starting prompt for every model. With it, Llama 4 Maverick 17B Instruct from Meta scored a Micro F1 of 0.75 on the development set. From there, the AWS team tuned each model’s prompt separately. For this model, v24 targeted room size overestimation, multi-label handling, and output format. Then v25, an automatically optimized version, moved the critical tasks first. In v27, MIPROv2 restructured the photo classification task, and the model’s Micro F1 on that task reached 0.76.
Where it ended
The final prompt, v27_PF, is v27 plus rules that Property Finder added for renders, images with borders or frames, and corridors. Like v27, it keeps the Arabic text rules and output format instructions. A normalization step after each model call handles remaining format differences.
With v27_PF, Llama 4 Maverick 17B Instruct scored a Micro F1 of 0.80 on the 356-image evaluation set, where proprietary model A scored 0.56 with v22. The evaluation set includes the 256 development images, and both models’ prompts were tuned on those images. These scores are therefore optimistic.
The switch from proprietary model A to Llama 4 Maverick 17B Instruct, each on its own tuned prompt, lifted Micro F1 from 0.55 to 0.81 on the development set. That is a 47 percent relative gain. Prompt tuning alone, on Llama 4 Maverick 17B Instruct, raised it from 0.75 with the shared v21 prompt to 0.81 with v27.
Lessons from failed approaches
Not everything succeeded. The 2-bucket approach put the watermark attributes and the other attributes in two separate API calls. It brought no meaningful accuracy gain. Boxed-image detection also dropped sharply. Without the watermark attributes in the same prompt, the models mistook watermark overlays for image borders.
Generic prompts also failed regardless of which model you ran them on. Real estate imagery has too much domain specificity for a one-size-fits-all instruction.
Results: Cost, accuracy, and speed
The AWS Generative AI Innovation Center team evaluated multiple models on Amazon Bedrock with the unbiased evaluation framework. All models started from the shared v21 prompt and were then tuned for their model family on the development set. Proprietary model A got one tuning round, v22. Llama 4 Maverick 17B Instruct got several, v24 to v27, plus Property Finder’s edge-case rules, so it had more tuning effort behind it. The final comparison against proprietary model A used the 356-image evaluation set. Property Finder selected Llama 4 Maverick 17B Instruct for their production workload based on accuracy across all 11 attributes and cost efficiency. For model availability by Region, refer to Supported models by AWS Region in Amazon Bedrock.
The following table shows the exact-match accuracy that Llama 4 Maverick 17B Instruct (v27_PF) reached on each single-label attribute, scored on the 356-image evaluation set. It scored 90 percent or higher on six of the nine, and matched or exceeded proprietary model A on all 11 attributes. On photo classification, which can have several labels per image, the two scored the same.
Cost per 1,000 images
Property Finder score
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.