看着
- 在企业规模上评价Microsoft Copilot需要测试数据，能够捕捉真实组织的全部复杂性，而真实的企业数据却包含在任何情况下都不能用于评价的敏感信息。
- 为解决这一紧张关系，我们建立了工业评估数据生成平台：以公开的实时数据来源为基础的全面合成企业环境，如O*NET职业数据、在线职位发布和证监会备案。
- 合成环境模仿真实的业务工作流程，帮助Microsoft找到通用的基准测试会错过的问题。
- 安全和隐私通过设计纳入平台。 每个环境都是完全合成和自动化的内容安全检查，使生成的数据没有有害或个人可识别的材料。
每天，全世界数百万知识工作者依靠Microsoft Copilot起草文件，总结会议，表面相关信息，并加快生产力。 在幕后，这是由人工智能系统网络提供动力的，确保这些系统如愿以偿，需要对评价采取深思熟虑的办法。 在Microsoft，我们知道高性能并非来自猜测。 其出自于测试，测量，并不断地延展。
医疗保健组织不断从各团队合成安全信号，以便他们能够做出更快，更好的决定，直接影响患者的结果和监管信心。
他们可以要求Copilot总结一次会面，为团队提取行动项目。 在数秒内，Copilot会发出一份结构化的总结，引用了笔录中的具体讨论要点。 但是，我们如何确保这种反应准确、完整并适当基于会议内容？这就是我们的评估系统。
我们制定了一个结构化的评价战略，称为工业评价数据生成平台，其中包括行业特定评价数据集，旨在对照现实的企业情景测试AI系统，其基础是三个核心支柱：
- 合成企业环境：模拟Microsoft Copilot环境，使用现实的用户、会议、电子邮件、文件和组织结构，而replicate则不使用任何客户数据，了解真实组织的复杂性
- 用于快速、可复制地创建评价数据集的自动查询生成管道
- 结构化质量框架，以确保合成数据符合进行有意义评价的标准
这些支柱共同帮助我们回答两个基本问题：Microsoft Copilot是否真的在做我们的用户需要它做的事，而我们能够在不接触他们的数据的情况下证明它？
从合成环境生成到自动查询管道
合成企业环境
评估企业AI需要测试数据，能够捕捉真实组织的全部复杂性：报告结构，项目历史，通信模式，以及特定领域的工作流程。 但真正的企业数据包含敏感信息，在任何情况下都不能用于评价。 这就是我们要解决的中心紧张局势。
我们不依靠手工设计的小型试验装置，而是产生完整的合成环境：模拟企业工作空间，每个工作空间都有现实的组织结构、交流历史和工作文物。 在Microsoft名词中，这些模拟工作空间中的每一个被称为"租户". 租户是一个孤立的组织环境，有自己的用户、数据和配置。 这些环境是可扩展的基线，各小组和组织可以随意建立。 它们旨在展示相同的结构和行为属性，使企业AI评价有意义。
以真实世界的信号定位
为了确保我们的合成环境是现实的，我们把它们建立在公开的可获取的真实世界数据来源中。
- 利用美国劳工部职业数据库的O*NET数据，使合成员工角色和责任现实化，调整分派给不同角色的工作活动的组合和频率，并调整行业特定工作流程，如临床试验审查或组合更新。
- 在线职位公布提供现实的角色头衔、报告等级和部门组成。
- 证监会的备案为组织规模、业务单位结构和财务背景提供信息。 所有这些来源均可公开查阅，并经过Microsoft法律和政策组织的审查和批准。
重要的是，这些合成环境不是静态快照。 它们模拟动态、时间变化的组织，包括会议历史、项目时间表、数周来发展起来的电子邮件线程以及自然展开的组织变化。 这种时间维度至关重要，因为大量的Copilot查询本身就具有时间意识。 “小组上周四决定了什么？” 或“自上个季度审查以来有什么变化？” 等问题只有在基本数据具有真正的时间结构时才会产生有意义的评价。
行业模拟
我们创造了三个合成环境，跨越多个行业，每个行业都有不同的特征，反映了真实世界的用法：
- 药品环境，包括监管提交工作流程、临床试验讨论和交叉功能安全审查会议
- 资本市场环境，包括交易管道、合规通信和客户组合审查
- 模拟短跑规划、代码审查讨论和产品发布协调的信息技术/软件公司
这种行业的特异性确保了我们的评价查询测试Copilot与这些域中真正的用户实际询问的各类问题，而不是依赖于可能不表面域特定故障模式的通用测试案例。
自动查询生成
虽然合成环境为评价提供了基础，但它们面临一个关键的限制：我们仍然需要现实的评价查询。 这些是检验Copilot能否表出正确的信息，正确合成，并有益地呈现出来的问题。 手工写上千个禁入，行业特定查询费用高得令人望而却步。 我们的自动输油管就是从那里进来的。
六阶段管道
我们建造了一个自动管道，将简单，通用的查询想法变成现实的，针对工业的Copilot问题。 并使用特定域的语言和背景来调整。 系统然后使用AI生成基于合成环境数据的查询，将查询重写为更自然和人性化的声音，并验证每个查询的准确性和独特性。 结果产生了一套丰富的高质量，活生生的查询，使得Copilot在现实世界的各种情景中能够被可伸缩和可靠地评价。
验证合成质量
当然，生成合成数据只是挑战的一部分。 我们还需要确保数据足以产生有意义的评价。 我们利用内部语义评价框架评估合成环境质量，该框架衡量三个层面的质量：
- 结构忠诚：组织结构是否反映了现实的报告等级、团队规模和跨职能关系？
- 关联性：查询和工作流程是否代表企业的实际使用模式？
- 行为现实主义：模拟用户是否展现出类似人类的交流模式和协作动态？
- 内容安全：生成的内容是否没有有害、攻击性或个人可识别的材料，是否避免冒充真正的个人？
端到端自动化
虽然我们的合成环境和查询管道提供了方法，但它们面临若干实际限制，没有适当的自动化，包括：
- 可复制性：人工程序每次产生不同的结果，难以比较整个发布的评价。
- 规模：行业垂直和配置变体的数量超过任何团队能够合理手工生产的数量。
- 迭接速度：人工创建需要几天时间，推迟基本反馈，并减缓发展周期。
- 跨小组一致性：没有标准化的工具，不同的团队会制作格式和质量不相容的评价数据集。
这种自动化减少了评价数据集的创建，从多日人工流程到自动运行。 我们估计，每个评价数据集可节省约100 000美元，项目总节省约300 000美元。 但更重要的结果是，任何团队现在都可以从同一个配置中重新生成一个评价数据集并获得相同的结果。
焦点：事件系列
隐私和设计安全
企业级别评价要求的不仅仅是良好的方法。 它需要一种隐私姿态，允许团队在不损害用户信任的情况下快速行动。 我国行业评价数据集的一个不可谈判的设计原则是在管道的任何阶段都不使用真正的客户数据。 从组织结构到电子邮件内容到会议记录，每个要素都是合成的，没有客户数据投入。 由于数据同时被完全合成和安全筛选，团队可以广泛共享并再利用，而无需引入声誉，合规，或责任AI风险。
我们的保障措施包括提供真实世界统计分布参考的隐私保护汇总技术，对所有公共数据来源进行专家法律审查和批准，在整个管道中明确数据来源和分类跟踪。
这意味着跨越Microsoft的团队可以自由地使用这些评价数据集进行实验，设定基准，并进行回归测试，而无需隐私审查间接费用或数据处理限制，否则会减缓迭代。 隐私设计并不是对系统的限制。 这正是使广泛通过成为可能的原因。
成果和影响
自该项目启动以来，我们看到在多个层面都有了重大改进。
在合成数据质量方面，我们的内部评分系统显示，在生成数据的结构、背景和统计可靠性方面，质量有了显著提高。 通过综合质量衡量标准(它衡量合成数据如何紧密地反映真实企业环境的复杂性和规律)，与我们最初的生成方法相比，得分提高了39%。 一些行业特定环境的质量分数超过1.0分的0.9分，表明域集中生成产生高度现实的评价环境。 这些改进部分是通过纳入现实的工业信号，部分是通过关键的复合质量衡量标准实现质量改进的自动化，这些衡量标准有助于改善我们合成工业环境的结构完整性、背景一致性和统计多样性。
在规模和效率方面，现在每个合成环境都包含数百个用户，拥有跨越三个行业纵向的完整通信历史。 评价数据集的建立从多日人工工作到管道自动运行。
在组织采纳方面，我国行业评价数据生成平台已发布内部共享目录，用于跨团队再利用，并被确定为Microsoft Copilot的第一个标准化评价基线。 不同之处不只是递增。 这是各小组如何对待评价的转变：从临时人工程序改为标准化
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
At a glance
- Evaluating Microsoft Copilot at enterprise scale requires test data that captures the full complexity of real organizations, yet real enterprise data contains sensitive information that cannot be used for evaluation under any circumstances.
- To resolve this tension, we built Industry Evaluation Data Generation Platform: full-scale synthetic enterprise environments grounded in publicly available real-world data sources such as O*NET occupational data, online job postings, and SEC filings.
- The synthetic environments mimic real business workflows, helping Microsoft find issues that generic benchmarks would miss.
- Safety and privacy are built into the platform by design. Every environment is fully synthetic and automated content-safety checks keep the generated data free of harmful or personally identifiable material.
Every day, millions of knowledge workers around the world rely on Microsoft Copilot to draft documents, summarize meetings, surface relevant information, and accelerate productivity. Behind the scenes, this is powered by a network of AI systems, and ensuring they work as intended requires a thoughtful approach to evaluation. At Microsoft, we know that high performance doesn’t come from guesswork. It comes from testing, measuring, and iterating constantly.
Healthcare organizations are constantly synthesizing safety signals from across teams so that they can make faster, better decisions that directly influence patient outcomes and regulatory confidence.
They may ask Copilot to summarize a meeting and extract action items for their teams. Within seconds, Copilot delivers a structured summary citing specific discussion points from the transcript. But how do we ensure this response is accurate, complete, and properly grounded in the meeting content? This is where our evaluation system comes in.
We’ve developed a structured evaluation strategy called Industry Evaluation Data Generation Platform, which includes industry specific evaluation datasets designed to test AI systems against realistic enterprise scenarios, grounded in three core pillars:
- Synthetic enterprise environments: simulated Microsoft Copilot environments with realistic users, meetings, emails, documents, and organizational structures that replicate the complexity of real organizations without using any customer data
- Automated query generation pipelines for rapid, reproducible creation of evaluation datasets
- Structured quality frameworks to ensure synthetic data meets the bar for meaningful evaluation
Together, these pillars help us answer two fundamental questions: Is Microsoft Copilot actually doing what our users need it to do, and can we prove it without ever touching their data?
In this blog post, we’ll walk through our evaluation system and processes, from synthetic environment generation to automated query pipelines, and how it all comes together to establish the first standardized evaluation baseline for Microsoft Copilot.
Synthetic enterprise environments
Evaluating enterprise AI requires test data that captures the full complexity of real organizations: reporting structures, project histories, communication patterns, and domain specific workflows. But real enterprise data contains sensitive information that cannot be used for evaluation under any circumstances. This is the central tension we set out to resolve.
Rather than relying on small, manually curated test fixtures, we generate full scale synthetic environments: simulated enterprise workspaces, each with realistic organizational structures, communication histories, and work artifacts. In Microsoft terminology, each of these simulated workspaces is called a “tenant”. A tenant is an isolated organizational environment with its own users, data, and configurations. These environments are scalable baselines that teams and organizations can build off of however they want. They are designed to exhibit the same structural and behavioral properties that make enterprise AI evaluation meaningful.
Grounding in real world signals
To ensure our synthetic environments are realistic, we grounded them in publicly available real world data sources.
- O*NET data from the U.S. Department of Labor’s occupational database was used to make synthetic employee roles and responsibilities realistic, calibrate the mix and frequency of work activities assigned to different roles, and tailor industry specific workflows such as clinical trial reviews or portfolio updates.
- Online job postings provide realistic role titles, reporting hierarchies, and departmental compositions.
- SEC filings inform organizational scale, business unit structures, and financial contexts. All of these sources are publicly available and have been reviewed and approved by Microsoft’s legal and policy organization.
Importantly, these synthetic environments are not static snapshots. They simulate dynamic, time-evolving organizations with meeting histories, project timelines, email threads that develop over weeks, and organizational changes that unfold naturally. This temporal dimension is essential because a large class of Copilot queries are inherently time aware. Questions like “What did the team decide last Thursday?” or “What’s changed since the last quarterly review?” only produce meaningful evaluations if the underlying data has a genuine temporal structure.
Industry specific simulation
We generated three synthetic environments across multiple industries, each with distinct characteristics that mirror real world usage:
- A pharmaceutical environment that includes regulatory submission workflows, clinical trial discussions, and cross functional safety review meetings
- A capital markets environment featuring deal pipelines, compliance communications, and client portfolio reviews
- An IT/software company that simulates sprint planning, code review discussions, and product launch coordination
This industry specificity ensures that our evaluation queries test Copilot against the kinds of questions real users in those domains actually ask, rather than relying on generic test cases that may not surface domain specific failure modes.
Automated query generation
While synthetic environments provide the substrate for evaluation, they face a key limitation: we still need realistic evaluation queries. These are the questions that test whether Copilot can surface the right information, synthesize it correctly, and present it helpfully. Manually writing thousands of grounded, industry specific queries is prohibitively expensive and slow. That’s where our automated pipeline comes in.
The six-stage pipeline
We built an automated pipeline that turns simple, generic query ideas into realistic, industry-specific questions for Copilot. It starts with a large library of common workplace tasks, narrows them down to what’s relevant for a given industry, and adapts them using domain-specific language and context. The system then uses AI to generate queries grounded in the synthetic environment’s data, rewrites them to sound more natural and human, and validates each one for accuracy and uniqueness. The result is a rich set of high-quality, lifelike queries that enable scalable and reliable evaluation of Copilot across real-world scenarios.
Validating synthetic quality
Of course, generating synthetic data is only part of the challenge. We also need to ensure the data is good enough to produce meaningful evaluations. We assess synthetic environment quality using our internal semantic evaluation framework which measures quality across three dimensions:
- Structural fidelity: Does the organizational structure reflect realistic reporting hierarchies, team sizes, and cross functional relationships?
- Contextual relevance: Are the queries and workflows representative of actual enterprise usage patterns?
- Behavioral realism: Do the simulated users exhibit human-like communication patterns and collaboration dynamics?
- Content safety: Is the generated content free of harmful, offensive, or personally identifiable material, and does it avoid impersonating real individuals?
End-to-end automation
While our synthetic environments and query pipelines provide the methodology, they face several practical limitations without proper automation, including:
- Reproducibility: Manual processes produce subtly different results each time, making it difficult to compare evaluations across releases.
- Scale: The number of industry verticals and configuration variants exceeds what any team can reasonably produce by hand.
- Iteration speed: Manual creation takes days, delaying essential feedback and slowing down development cycles.
- Cross-team consistency: Without standardized tooling, different teams produce evaluation datasets with incompatible formats and quality levels.
This automation reduced evaluation dataset creation from a multi-day manual process to automated runs. We estimate savings of approximately $100,000 per evaluation dataset and $300,000 in total project savings. But the more important outcome is that any team can now regenerate an evaluation dataset from the same configuration and get identical results.
Spotlight: Event Series
Privacy and safety by design
Enterprise grade evaluation demands more than good methodology. It requires a privacy posture that allows teams to move fast without compromising user trust. A non-negotiable design principle of our industry evaluation datasets is that no real customer data is used at any stage of the pipeline. Every element, from organizational structures to email content to meeting transcripts, is synthetically generated, with no customer data inputs. Because the data is both fully synthetic and safety-screened, teams can share and reuse it widely without introducing reputational, compliance, or Responsible AI risk.
Our safeguards include privacy preserving aggregation techniques where real-world statistical distributions are referenced, expert legal review and approval of all public data sources, and clear data provenance and lineage tracking throughout the pipeline.
This means teams across Microsoft can use these evaluation datasets freely for experimentation, benchmarking, and regression testing without privacy review overhead or data handling restrictions that would otherwise slow down iteration. The privacy design is not a constraint on the system. It is what makes broad adoption possible.
Results and impact
Since starting this project, we have observed significant improvements across multiple dimensions.
On synthetic data quality, our internal scoring system showed significant quality gains in the structural, contextual, and statistical fidelity of the generated data. Across our composite quality metric (which measures how closely synthetic data mirrors the complexity and patterns of real enterprise environments), scores improved 39% compared to our initial generation approach. Some industry-specific environments reached quality scores above 0.9 out of 1.0, demonstrating that domain focused generation produces highly realistic evaluation environments. The improvements were obtained in part by incorporating realistic, industry signals and in part by automating quality improvements via key composite quality metrics that help improve the structural integrity, contextual coherence and statistical diversity of our synthetic industry environments.
On scale and efficiency, each synthetic environment now contains hundreds of users with complete communication histories across three industry verticals. Evaluation dataset creation has gone from multi-day manual effort to automated pipeline runs.
On organizational adoption, our industry evaluation data generation platforms have been published to an internal shared catalog for cross-team reuse and established as the first standardized evaluation baseline for Microsoft Copilot. The difference wasn’t just incremental. It was a shift in how teams approach evaluation: from ad hoc, manual processes to standardized,
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.