AWS Batch的配额管理为SageMaker培训工作提供跨团队借款和资源

人工智能

AWS Batch的配额管理为SageMaker培训工作提供跨团队借款和资源

由于AWS批次的发布支持了Amazon SageMaker训练工作，客户获得了一种新的全管理方式来排队，排出，并重试机器学习(ML)训练工作量而不建立自己的基础设施。 相反，他们使用AWS批次的排队和排程功能。 随着各组织扩大人工智能和机器学习(AI/ML)业务的规模并增加机上团队，协调共享图形处理单元(GPU)能力的访问，可以为管理员带来额外的通信、协调和配置间接费用。 这种间接费用包括设置单独的排队，调整容量限制，以及精心安排合适的工作量，以在合适的时间获得合适的资源。 AWS Batch的一个核心价值命题是帮助客户增加其计算利用率，随着培训环境的增长，利用现有能力做更多工作的机会也随之增加。

从客户管理多团队训练环境向后工作，我们听到一个一致的要求：给我们一个方法来定义每个团队或项目的拨款，自动分享闲置能力，并首先管理我们最优先的工作量。 这一请求导致我们建立配额管理，这是AWS批次的新排程能力。 管理员可以使用它来将计算能力分为定额份额，按团队，项目或工作量来组织。 每个配额份额都采用可配置的资源共享战略，最大限度地利用GPU，同时尊重每个团队最重要的分配。

这篇文章显示配额管理如何运作，包括如何用出借和借贷策略配置配额份额，为优先工作量设定共享优先，并开始使用"SageMaker Python SDK".

配额份额和预扣

管理员可以通过为单个团队或项目等单独的实体规定配额份额来分配并共享受限制的GPU资源。 每个配额份额包括计算配额(容量限制)和闲置计算的资源共享策略。 这些策略决定了闲置计算是保留到工作到来(RESERVE)，借给有工作的其他配额份额(LEND)，还是在闲置和借出该份额的工作可能有利时借出(LEND AND BORROW).

配额份额作为虚拟队列在关联的任务队列内依次嵌入。 在为工作队列排出任务时，AWS Batch优先处理在通过相联的配额份额递增时处于其容量限制下的配额份额。 如果工作达到一定的配额份额，则AWS批量将进行跨份额优先。 如此一来，管理者就不需要追踪工作借款能力，AWS批次检查是否有从另一批配额借款的能力。 如果是的话，工作量可以使用其他零散的能力来承担。

您可以在配置配额时打开基于优先级的共享预置。 交叉股权先发制人在工作达到贷款配额时恢复所借能力，分权先发制人则先发制人，在计划D、STARTING或RUNNING状态下降低优先工作。 这使得队列前方的更高优先级的工作可以更早地开始运行。

先决条件

在此示例之前，您必须拥有 AWS 账户，并有权限创建和管理 AWS 批量资源。 举例来说，请参看引用AWS Batch和SageMaker APIS所需的IAM权限样本。

例子使用 SageMaker Python SDK v3. 完成安装相容版本 SageMaker Python SDK 所需的设置：

配置配额管理资源。

利用新的配额管理能力需要制定包含配额共享政策的时间安排政策、规定其具体实例能力限度的服务环境，以及与两者相连的SageMaker培训工作队列。 之后，您可以将配额股份与工作队列联系起来。 用户指南有AWS指令行接口(AWS CLI)和配置配额管理资源和配额份额的控制台实例。 SageMaker Python SDK v3中与aws batch模块整合的从端到端配额管理实例被包含在一个样本笔记本中。 在这个例子中，我们跨出代码片段来总结关键组件，并使用笔记本中的帮助器功能来创建一个调度策略，服务环境，工作队列，以及三个配额共享。

在此配置的三个配额份额代表三个不同的项目。 每一个都有相同的容量限制：一个ml.g5.x大实例。 QS1配额份额代表背景工作。 对于这个项目，您想要先进行最重要的背景工作，并尽可能地借用可用的剩余容量。 在这个配额份额内，没有紧迫的工作，而且由于所有的工作都使用检查站，所以先发制人是可以接受的。 这个配额份额使用 LEND AND BORROW 资源共享策略，并开启了共享优先级。 在同一配额范围内的较高优先工作可以预先阻止优先程度较低的工作。

QS2是中等优先的工作量。 选择这种配置的团队宁愿等待能力到位，而不是忍受额外的先发制人。 这些先发制人来源于借来的能力被收回，或者由于较优先的工作申请而被分享。 这适用于较短的工作，或大多数提交的工作具有类似的优先次序。 配额份额仍然受益于在派遣工作时所考虑的工作优先事项。 提交的工作触发了AWS批量，以收回任何用于管理工作所需的出借能力。

QS3是需要以尽可能少的中断时间尽快开始的工作量。 它的配置是为了保留能力，既不向其他配额提供贷款，也不向其他配额提供贷款。 因为它的工作量没有超过它们的能力限度。 通过同时关闭股份预置，提交这一配额份额的工作量不会被AWS批次预先控制，只要容量限制不变。

这些配额共享配置是对样本笔记本中创建 quota management resources函数的输入，该功能还配置了相应的列表策略，服务环境，以及任务队列，并带有所选择的恒定名称。

向 QS1 提交 SageMaker 培训任务

SageMaker Python SDK内部的aws batch模块已经扩展，允许将SageMaker培训任务的程序提交到一个配额管理驱动的AWS Batch任务队列。 我们使用Hello World ModelTrainer，以下代码片段根据所选择的区域和实例类型确定要使用的图像 URI。

我们用这个图像 URI 在一个基本的问候世界模型教练的例子中：

要在 SageMaker Python SDK 内使用任务队列，请在 aws batch 模块内为训练队列构造器提供队列名称。

然后，我们可以向QS1提交三个工作，我们的背景工作配额份额。 提交的第一份工作具有最高优先级，其次是中等优先级，然后是最低优先级。 由于QS1或QS2目前没有运行任何工作，两例工作的价值应该开始，与提交的第一和第二个工作相对应(因为它们具有最高的相对优先地位)。 请注意，由于这仅仅是一个问候性的世界培训工作，它并不需要数据，而是典型的模型训练员需要数据来进行培训。

将工作提交QS2触发跨份额先发制人以收回出借的能力

拥有QS2的团队可能发现他们有工作来表演，并向QS2提交工作。 如此一来触发了QS1对容量的先发制人，而QS1正在借用这一容量(QS3的容量没有被配置为被借用). 此是取相分相先入相。

更新QS1中的工作优先级触发共享优先级，以管理最高优先级的工作

如果科学家意识到QS1中的 qs1 job low job 在 QS1 中仍然可以运行，由于优先事项的变化，现在需要开始运行，他们可以使用新的 UpdateServiceJob API来提高已经提交的 QS1 中的工作的优先级。 因为没有可用的借出能力，所以这可以启动共享先发制人，这样当前运行的工作就可以被先发制人，Qs1 job low就可以开始。

(备选) 清理已创建的资源

为清理此帖子所创造的资源，运行以下代码：

结论

以及如何与SageMaker训练工作一起使用，并开始为自己的项目设定配额份额。 详见AWS批量配额管理用户指南。

AWS 时速配有配额管理可以帮助你的组织提高计算利用率并更有效地优先安排你的团队和工作量，为你的基础设施管理员和ML科学家腾出时间。 通过实行配额管理，你们的团队可以专注于工作量，并减少管理和协调基础设施所花的时间。 这种能力在Amazon SageMaker AI的培训计划中特别强大，这样你的组织就可以在需要的时候按所需数量保留能力。 通过使用与SageMaker AI的AWS批量，可以充分利用训练计划。 我们鼓励你们尝试这种新能力，以便在你们的行动中产生有意义的影响。

本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

Artificial Intelligence

Lend and borrow resources across teams with AWS Batch’s quota management for SageMaker Training jobs

With the launch of AWS Batch support for Amazon SageMaker Training jobs, customers gained a new fully managed way to queue, schedule, and retry machine learning (ML) training workloads without building their own infrastructure. Instead, they use the queuing and scheduling features of AWS Batch. As organizations scale their artificial intelligence and machine learning (AI/ML) operations and onboard more teams, coordinating access to shared graphics processing unit (GPU) capacity can introduce additional communication, coordination, and configuration overhead for administrators. This overhead includes setting up separate queues, adjusting capacity limits, and carefully orchestrating the right workloads to get the right resources at the right time. A core value proposition of AWS Batch is helping customers increase utilization of their compute, and as training environments grow, so does the opportunity to do more with existing capacity.

Working backward from customers managing multi-team training environments, we heard a consistent request: give us a way to define allocations per team or project, share idle capacity automatically, and run our highest-priority workloads first. This request led us to build quota management, a new scheduling capability for AWS Batch. Administrators can use it to divide compute capacity into quota shares, organized by team, project, or workload. Each quota share uses configurable resource-sharing strategies that maximize GPU utilization while honoring each team’s allocation when it matters most.

This post shows how quota management works, including how to configure quota shares with lending and borrowing strategies, set up in-share preemption for priority workloads, and get started using the SageMaker Python SDK.

Quota shares and preemption

Administrators can allocate and share constrained GPU resources by defining a quota share for separate entities, such as individual teams or projects. Each quota share includes compute quotas (capacity limits) and resource-sharing strategies for idle compute. These strategies determine whether idle compute is reserved until jobs arrive (RESERVE), lent to other quota shares that have work available (LEND), or both lent when idle and borrowed when the share has work that might benefit (LEND_AND_BORROW).

A quota share operates as a virtual queue nested hierarchically within an associated job queue. When scheduling jobs for a job queue, AWS Batch prioritizes quota shares that are under their capacity limits as it iterates through the associated quota shares. If jobs arrive in a quota share that has lent out its capacity, AWS Batch performs cross-share preemption. This reclaims resources for the lending quota share, so administrators don’t need to track down the jobs borrowing capacity and terminate them on their own. If jobs arrive in a quota share that is at or above its capacity limit, AWS Batch checks whether there’s available capacity to borrow from another quota share. If so, workloads can run using otherwise-idle capacity.

You can turn on priority-based in-share preemption when you configure a quota share. Where cross-share preemption reclaims lent capacity as jobs arrive in a lending quota share, in-share preemption preempts lower priority jobs that are in the SCHEDULED, STARTING, or RUNNING state. This allows higher priority jobs at the front of the queue to start running sooner.

Prerequisites

Before you proceed with this example, you must have an AWS account with permissions to create and manage AWS Batch resources. For this example, see the sample IAM permissions needed to invoke AWS Batch and SageMaker APIs.

The example uses the SageMaker Python SDK v3. Complete the setup required to install a compatible version of the SageMaker Python SDK:

Configuring quota management resources

Using the new quota management capabilities requires creating a Scheduling Policy that contains a quotaSharePolicy, a Service Environment that specifies its capacity limits in terms of specific instances, and a SageMaker Training job queue that is connected to both. After that, you can associate quota shares to a job queue. The user guide has AWS Command Line Interface (AWS CLI) and console examples for configuring quota management resources and quota shares. An end-to-end example of quota management integrated with the aws_batch module in the SageMaker Python SDK v3 is contained in a sample notebook. In this example, we step through code snippets that summarize the key components, and use the helper functions from the notebook to create a Scheduling Policy, Service Environment, Job Queue, and three Quota Shares.

The three quota shares configured here represent three different projects. Each has the same capacity limit set: a single ml.g5.xlarge instance. The QS1 quota share represents background jobs. For this project, you want to run the most important background job first and borrow as much spare capacity as is available. Within this quota share, no jobs are urgent, and preemptions are acceptable because all jobs use checkpointing. This quota share uses the LEND_AND_BORROW resource-sharing strategy and has in-share preemption turned on. Higher priority jobs within the same quota share can then preempt running jobs that are lower priority.

QS2 represents medium priority workloads. Teams choosing this configuration would prefer to wait for capacity to become available rather than endure additional preemptions. Those preemptions come from borrowed capacity being reclaimed or from in-share preemptions because of higher priority job submission. This is appropriate for shorter jobs, or when most submitted jobs have similar priorities. The quota share still benefits from job priority being taken into consideration at job dispatch. Submitted jobs trigger AWS Batch to reclaim any lent capacity needed to run the jobs.

QS3 represents workloads that need to start as quickly as possible with minimal interruptions. It’s configured to reserve capacity and neither lends nor borrows to other quota shares. It doesn’t experience cross-share preemptions, because its workloads don’t exceed their capacity limits. By also turning off in-share preemption, workloads submitted to this quota share are not preempted by AWS Batch as long as the capacity limit is unchanged.

These quota share configurations are inputs to the create_quota_management_resources function in the sample notebook, which also configures the corresponding Scheduling Policy, Service Environment, and Job Queue with the constant names chosen.

Submit SageMaker Training jobs to QS1

The aws_batch module within the SageMaker Python SDK has been extended to allow programmatic submission of SageMaker Training jobs to a quota-management-enabled AWS Batch job queue. To demonstrate these new capabilities, we use a Hello World ModelTrainer that’s submitted to the job queue. The following code snippet identifies an image URI to use based on the Region and instance type selected.

We use this image URI within a basic hello world model trainer example:

To use the job queue within the SageMaker Python SDK, provide the queue name to the TrainingQueue constructor within the aws_batch module.

We can then submit three jobs to QS1, our background jobs quota share. The first job submitted has the highest priority, followed by the medium priority job, and then the lowest priority job. As nothing is currently running in QS1 or QS2, two instances’ worth of jobs should start, corresponding to the first and second jobs submitted (as they had the highest relative priority). Note that because this is only a hello world training job, it doesn’t require data, but typically a ModelTrainer requires data to perform training.

Submitting jobs to QS2 triggers cross-share preemption to reclaim lent capacity

The team that owns QS2 might discover they have work to perform and submit a job to QS2. Doing so triggers a preemption of capacity back from QS1, which is currently borrowing that capacity (QS3’s capacity isn’t configured to be borrowed). This is a cross-share preemption.

Updating job priorities in QS1 triggers in-share preemption to run highest priority jobs

If a scientist realizes that the qs1_job_low job in QS1, which is still RUNNABLE, needs to begin running now because of changing priorities, they can use the new UpdateServiceJob API to increase the priority of an already submitted job in QS1. Because there’s no available capacity to borrow, this kicks off an in-share preemption, so that the job currently running is preempted and qs1_job_low can begin.

(Optional) Clean up created resources

To clean up the resources created in this post, run the following code:

Conclusion

In this post, we covered quota management, a new scheduling capability in AWS Batch, and how to use it with SageMaker Training jobs and get started setting up quota shares for your own projects. For more details, see the AWS Batch quota management user guide.

AWS Batch with quota management can help your organization increase compute utilization and more effectively prioritize your teams and workloads, freeing up time for your infrastructure admins and ML scientists. By implementing quota management, your teams can focus on their workloads and reduce time spent managing and coordinating infrastructure. This capability is especially powerful with Amazon SageMaker AI training plans, so that your organization can reserve capacity in the quantity you need, during the time you need it. By using AWS Batch with SageMaker AI, you can make full use of the training plan. We encourage you to try out this new capability so it can make a meaningful impact in your operations.

Auto-fetched and translated by AI Gem Box. All rights belong to the original author.