人工智能
使用 Amazon Nova 法案-第 3 部分的代理 QA 自动化加速软件交付
随着质量保证自动化的采用在一个组织内部不断增长，一个与写作测试无关的新问题似乎已经出现。 一个平台团队站起测试自动化，然后第二个团队想加入，然后再加入一打。 很快有上百个测试案例属于不同的产品和团队，每个人都可以看到其他人的测试。 没有人可以一眼就回答一个基本问题：这个应用程序的测试覆盖率现在是否健康？测试作者不再是瓶颈组织测试和控制谁能触摸它们，就成了工作。
第二个相关问题紧随其后：该应用程序及其测试由谁拥有，在覆盖范围缩小时由谁负责？没有明确的主人，健康问题就没有人可以降落。
在第一篇文章中，我们推出 QA Studio，一个用Amazon Nova Act所建的参考解决方案。 团队用它来定义自然语言的测试，并通过AI动力的视觉导航运行。 在第二篇文章中，我们增加了测试套件和一个CLI，用于连续集成和连续交付(CI/CD)管道中的批量回归测试。 协助QA Studio跨组织规模。 首先是应用、组织测试和报告健康指标。 第二，基于团队的用户管理，控制谁能看到和改变什么。
组织应用测试
一个QA Studio中的应用程序代表测试下的系统：一个网站，一个网络应用，一个服务前端。 是容器收集了测试资料、住宅和业主。 使用案例(来自第1部分的个别测试)和测试套件(来自第2部分的回归集合)属于一个应用程序，它们继承其所有权。 每支队伍在部署中的每一次测试都有一个平面列表，而不是在自己的应用程序内工作。
创建应用程序需要一个名称、一个基础URL以及一组可选的环境，如开发、中转和生产。 从应用程序页面中，团队直接在上下文中创建出新的用例，并添加出已存在的用例，将其分组入测试套件。 这也是QA Studio报告其进行测试的健康状况的地方。
以衡量标准了解应用健康
试验产生结果。 随着时间的推移，了解这些结果会告诉一个团队应用的开发生命周期是否健康，而不只是单个建筑是否准备释放。 QA Studio将每次执行的结果记录为Amazon Cloud Watch自定义度量衡，以应用和环境为维度。 它将这些衡量标准转化为应用页面的健康视图。
页面上方以三个数字解答了眼前的问题：所选窗口的通过率，处决总数，以及目前的失败计数。 QA Studio计算健康得分为0-100。 得分从通过率开始，并减去对片状测试的处罚。 如此一来，考试通过却在通过与失败之间翻转的应用程序，一个团队扫描一组应用程序，就可以分辨出那些需要注意的应用程序，而不打开每个结果。
度量衡标签从当前快照移动到趋势。 团队选择一个有小时颗粒度的7天窗口或有每日颗粒度的30天窗口，QA Studio则在这段时间里绘制了处决，成功，失败以及平均持续时间。 由于基本衡量标准带有环境层面，团队将视图过滤到单一环境。 例如，它们可以比较同一套试验的生产相比的组装行为。
两种观点针对的是那些成本最高的团队的测试。 片面测试以过去七天中，这揭示了间歇性故障，这些故障侵蚀了对套房的信任而后演变为出产错。 近期的失败视图列出了最近因快速分解而被处决的失败情况。 每个都链接到第1部分描述的Amazon Nova Act轨迹记录和会话录音。 这样，从“这一测试失败”到“agent所看到的情况”只需一击。
控制与团队和角色的接触
量子告诉一个团队他们的应用是如何做的。 基于团队的用户管理层首先决定哪些应用程序是他们的。 QA Studio在团队周围组织访问，一个应用程序完全属于一个团队。 该团队成员处理其申请。 其他人看不到他们。
每名队员各有五个角色之一，每个角色包括下面的队员的许可：
- 管理员：全工作室访问每个团队、应用程序和用户，包括创建团队和分配应用程序。 Admin是一个全工作室的角色，而不是每个团队的任务。
- 团队领导：管理自己的团队成员并为团队创建应用程序。
- 编辑：在团队的应用程序中创建和编辑使用案例和测试套件。
- 执行者：运行使用大小写和套件，但不修改。
- 查看器：只读访问团队的应用程序和结果。
由于角色是累积的，一个团队领跑者拥有编辑器拥有的一切，等等，下到Viewer. 使用例或测试套件的取用遵循包含该套件的应用程序。 如果一个成员可以从事申请工作，他们可以从事测试工作，并以他们的角色为界限。 这使许可管理保持在应用程序层面，而不是在数千个个人测试上。
一个管理员看到整个工作室。 用户页面列出每个账户及其团队和角色在一个地方。 行政主管可以知道谁拥有团队领导权，谁拥有全工作室管理员的权利。 在此处显示的参考部署中，四个用户跨越两个小组：两个小组领导、一个查看员和一个全工作室管理员。 团队的页面逐个滚动同样的数据，显示每个团队的成员数和应用程序数并列。 管理员可以在打开其中任何一个之前看到哪些球队正在扩大。
由小组滚动的同一成员数据见小组网页。
从团队切换到应用程序，整个工作室的应用程序列出了所有团队的每个应用程序的显示。
团队间共享应用程序
团队很少完全孤立地工作。 平台团队可能拥有一个功能团队需要运行来对抗的应用程序，而无需交出更改它的能力。 QA Studio通过共享访问支持这一点：拥有团队在视图，执行或完整关卡允许另一团队访问一个应用程序。
共享使用遵循上限规则。 用户对共享申请的有效许可是其自身角色和补助金水平的较低。 被允许充分访问共享应用程序的查看器仍然只查看，因为他们的作用是上限。 这样一来，一个拥有的团队就可以广泛分享，而不给予一个接受者所允许的更多角色。
一支队伍所看到的
一个团队领导在一个团队内部工作，而不是整个工作室。 “开放我的团队”提供了四个标签，涵盖所有主要控制：成员、应用、使用案例和共享访问。 成员列表列出团队中的人员。 领导者可以改变任何成员的角色，或者增加一个新的角色，因此日常访问管理会留在最接近工作的人身边。 在此处显示的AWS团队中，领跑者管理着两名成员，一名是团队领跑者和一名取景者。
应用程序标签显示只包含团队拥有的应用程序，每个应用程序都有其测试大小写计数和最后运行时间。 这使得主角一直专注于自己的表面，而不是完整的工作室库存。 共享访问标签是跨团队共享成为混凝土的地方。 在此，AWS Team已经允许AMZN Team查看一个应用程序，领先者可以批准，撤销，或改变访问级别。 由于赠款为收视水平，一个AMZN团队编辑仍然只看共享应用程序：上一节的上限规则，在实践中适用。
同一团队领队的共享访问标签记录其他团队可以到达团队的应用程序，在哪个级别。
如何运作
QA Studio在先前帖子描述的同一种无服务器的地基上建立基于团队的访问控制和度量衡，在混音中增加了一些AWS服务：
- Amazon Cognito：每个团队向一组Cognitto组地图，每个角色一个。 AWS Lambda授权人每次请求都会将呼叫者的团体解入API范围和团队背景。
- Amazon DynamoDB：团队，成员，应用程序所有权，以及跨团队共享赠款被存储在现有单表设计中。
- Amazon Cloud Watch：每个执行都发出QAStudio/Executions命名空间中自定义的度量衡，以应用和环境为维度。 API读取它们来构建健康观点和趋势图。
- AWS Lambda和Amazon API Gateway：后端在两个层次上强制访问：一个是呼叫者角色授予的范围，另一个是确认呼叫者团队拥有或已经与应用程序共享的资源检查。
由于访问来源于Cognitto集团会籍，OAuth客户端证书出自Part 2，符合同一种模式。 一个CI管道作为机器身份认证具有它需要的瞄准镜，不再有。
开始
应用程序，度量衡和基于团队的访问控制是QA Studio参考解决方案的一部分，您在自己的AWS账户中用AWS Cloud Development Kit(AWS CDK)部署。 所有测试数据、结果和度量度都位于您的账户范围内。 部署后，管理员会创建团队，指派成员和角色，并向团队分配应用程序。 每个团队的健康标准随着测试的进行而开始流行。
关于部署指令和配置细节，参见QA Studio GitHub寄存器及其文档。
结论
在本文中，我们展示了QA Studio如何将代理QA自动化从单个测试到一个组织的规模。 申请给每个团队一个使用案例和测试套房的家。 他们通过通过通过率、趋势图、微分调整后的健康分数和基于Amazon Cloud Watch 测量的故障分数报告健康情况。 基于团队的用户管理由Amazon Cognitto支持，让每个团队在自己的应用程序内工作，跨团队共享以明确的上限规则为限制。
这些能力建立在第一编自然语言测试的作者和第二编的套件和CI/CD集成的基础上。 它们共同帮助QA自动化在众多团队中成长而不会失去能见度或控制。 为了在自己的环境中尝试，从QA Studio GitHub寄存器中部署参考解决方案，创建了您的第一个团队和应用程序，并运行了少数使用的案例来观察公尺成形。 要了解更多背后的agent，请参见Amazon Nova Act产品页。
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Artificial Intelligence
Accelerating software delivery with agentic QA automation using Amazon Nova Act – Part 3
As quality assurance (QA) automation adoption grows inside an organization, a new problem appears that has nothing to do with writing tests. A platform team stands up test automation, then a second team wants to join, then a dozen more. Soon hundreds of test cases belong to different products and squads, and everyone can see everyone else’s tests. No one can answer a basic question at a glance: is this application’s test coverage healthy right now? Test authoring stops being the bottleneck. Organizing tests and controlling who can touch them becomes the work.
A second, related question follows close behind: who owns this application and its tests, and who is accountable when coverage slips? Without a clear owner, health problems have no one to land on.
In the first post, we introduced QA Studio, a reference solution built with Amazon Nova Act. Teams use it to define tests in natural language and run them through AI-powered visual navigation. In the second post, we added test suites and a CLI for batch regression testing in continuous integration and continuous delivery (CI/CD) pipelines. In this post, we cover two capabilities that help QA Studio scale across an organization. The first, applications, organizes tests and reports health metrics. The second, team-based user management, controls who can see and change what.
Organizing tests with applications
An application in QA Studio represents the system under test: a website, a web app, a service front end. It’s the container that gives a collection of tests, a home, and an owner. Use cases (the individual tests from Part 1) and test suites (the regression collections from Part 2) belong to an application, and they inherit its ownership. Instead of one flat list of every test in the deployment, each team works inside the applications they own.
Creating an application takes a name, a base URL, and an optional set of environments such as development, staging, and production. From the application page, teams create new use cases directly in context, add existing ones, and group them into test suites. This is also where QA Studio reports on the health of the tests it runs.
Understanding application health with metrics
Tests produce results. Understanding those results over time is what tells a team whether the development lifecycle for an application is healthy, not only whether a single build is ready for release. QA Studio records the outcome of every execution as Amazon CloudWatch custom metrics, dimensioned by application and by environment. It turns these metrics into a health view on the application page.
The top of the page answers the immediate question with three figures: the pass rate over the selected window, the total number of executions, and the current failure count. Alongside them, QA Studio computes a health score of 0–100. The score starts from the pass rate and subtracts a penalty for flaky tests. That way, an application whose tests pass but flip between pass and fail doesn’t present a falsely perfect picture. A team scanning a portfolio of applications can tell the stable ones from the ones that need attention without opening each result.
The metrics tab moves from the current snapshot to trends. Teams choose a 7-day window with hourly granularity or a 30-day window with daily granularity, and QA Studio charts executions, successes, failures, and average duration across that period. Because the underlying metrics carry an environment dimension, teams filter the view to a single environment. For example, they can compare how staging behaves against production for the same set of tests.
Two views target the tests that cost teams the most time. The flaky tests view ranks use cases by how often they flipped between passing and failing over the last seven days. This surfaces the intermittent failures that erode trust in a suite before they turn into a production miss. The recent failures view lists the latest failed executions for quick triage. Each one links back to the Amazon Nova Act trajectory logs and session recordings described in Part 1. That way, moving from “this test is failing” to “here is exactly what the agent saw” takes one click.
Controlling access with teams and roles
Metrics tell a team how their applications are doing. Team-based user management decides which applications are theirs in the first place. QA Studio organizes access around teams, and an application belongs to exactly one team. Members of that team work with its applications. Everyone else doesn’t see them.
Every team member holds one of five roles, and each role includes the permissions of the ones below it:
- Admin: Studio-wide access to every team, application, and user, including creating teams and assigning applications. Admin is a studio-wide role rather than a per-team assignment.
- Team Lead: Manages their own team members and creates applications for the team.
- Editor: Creates and edits use cases and test suites within the team’s applications.
- Executor: Runs use cases and suites but doesn’t modify them.
- Viewer: Read-only access to the team’s applications and results.
Because roles are cumulative, a Team Lead has everything an Editor has, and so on down to Viewer. Access to a use case or a test suite follows the application that contains it. If a member can work with the application, they can work with its tests, bounded by their role. This keeps permission management at the application level instead of on thousands of individual tests.
An administrator sees the whole studio. The Users page lists every account with its team and role in one place. An admin can tell who holds Team Lead on which team and who carries Studio-wide Administrator rights. In the reference deployment shown here, four users span two teams: two Team Leads, one Viewer, and one Studio-wide Administrator. The Teams page rolls the same data up by team, showing each team’s member count and application count side by side. An admin can see which teams are expanding before opening any one of them.
The same membership data, rolled up by team, appears on the Teams page.
Switching from teams to applications, the studio-wide Applications list shows every application across all teams.
Sharing applications between teams
Teams rarely work in complete isolation. A platform team might own an application that a feature team needs to run against, without handing over the ability to change it. QA Studio supports this with shared access: the owning team grants another team access to an application at a view, execute, or full level.
Shared access follows a ceiling rule. A user’s effective permission on a shared application is the lower of their own role and the level of the grant. A Viewer who is given full access to a shared application remains view-only, because their role is the ceiling. This way, an owning team can share widely without granting more than a recipient’s role allows.
What a team lead sees
A team lead works inside a single team rather than the whole studio. Opening My Team presents four tabs that cover everything the lead controls: Members, Applications, Use Cases, and Shared Access. The Members tab lists the people on the team. The lead can change any member’s role inline or add a new one, so day-to-day access management stays with the person closest to the work. In the AWS Team shown here, the lead manages two members, one Team Lead and one Viewer.
The Applications tab shows only the applications the team owns, each with its test-case count and last-run time. This keeps the lead focused on their own surface rather than the full studio inventory. The Shared Access tab is where cross-team sharing becomes concrete. Here the AWS Team has granted the AMZN Team view access to an application, and the lead can grant, revoke, or change the level of that access. Because the grant was view level, an AMZN Team Editor still sees the shared application read-only: the ceiling rule from the previous section, applied in practice.
The same Team Lead’s Shared Access tab records which other teams can reach the team’s applications, and at what level.
How it works
QA Studio builds team-based access control and metrics on the same serverless foundation described in the earlier posts, adding a few AWS services to the mix:
- Amazon Cognito: Each team maps to a set of Cognito groups, one per role. An AWS Lambda authorizer resolves a caller’s groups into API scopes and team context on every request.
- Amazon DynamoDB: Teams, memberships, application ownership, and cross-team share grants are stored in the existing single-table design.
- Amazon CloudWatch: Every execution emits custom metrics in the QAStudio/Executions namespace, dimensioned by application and environment. The metrics API reads them back to build the health view and trend charts.
- AWS Lambda and Amazon API Gateway: The backend enforces access at two layers: the scope a caller’s role grants, and a resource check that confirms the caller’s team owns or has been shared with the application.
Because access is derived from Cognito group membership, the OAuth client credentials flow from Part 2 fits the same model. A CI pipeline authenticates as a machine identity with the scopes it needs and no more.
Getting started
Applications, metrics, and team-based access control are part of the QA Studio reference solution, which you deploy in your own AWS account with the AWS Cloud Development Kit (AWS CDK). All test data, results, and metrics stay within your account boundaries. After deploying, an administrator creates teams, assigns members and roles, and assigns applications to teams. Each team’s health metrics begin populating as their tests run.
For deployment instructions and configuration details, see the QA Studio GitHub repository and its documentation.
Conclusion
In this post, we showed how QA Studio scales agentic QA automation from individual tests to an organization. Applications give every team a home for their use cases and test suites. They report health through pass rate, trend charts, a flaky-adjusted health score, and failure triage built on Amazon CloudWatch metrics. Team-based user management, backed by Amazon Cognito, keeps each team working within the applications they own, with cross-team sharing bounded by a clear ceiling rule.
These capabilities build on the natural language test authoring from Part 1 and the suites and CI/CD integration from Part 2. Together, they help QA automation grow across many teams without losing visibility or control. To try it in your own environment, deploy the reference solution from the QA Studio GitHub repository, create your first team and application, and run a few use cases to watch the metrics take shape. To learn more about the agent behind it, see the Amazon Nova Act product page.
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.