人工智能
K圆圈如何加速 DevOps 事件分解到 15分钟以下与 Amazon Bedrock
本文由Cloud DevOps Engineer at Circle K共同撰写。
许多DevOps团队都知道事件的分门别类模式：警报起火，在任何人都能解决问题之前，一名工程师花费了45-60分钟的时间去弄清楚发生了什么。 他们抽取日志，检查服务健康情况，以及一些工具的相关症状。 所有这些发生在补救开始之前。
在Circle K，这个挑战随着业务的展开而扩大。 该公司在27个国家经营超过17,200个便利和燃料零售店。 在店面后，一个云母基础设施支持售出点系统，忠诚应用，以及运行于AWS上的电动车辆充电应用。 随着公司扩大数字业务，分级管理费随其增长而增加。 更多的服务意味着更多的警报，在开始补救之前，每个部门都需要同样的人工调查。
为了解决这个问题，Circle K使用Amazon Bedrock构建了自动分录系统，这是一种完全管理的服务，通过单一的API提供基础模型(FM). 对于由AWS区域提供的模型可用性，指由AWS区域在Amazon Bedrock中支持的模型。 系统使用代理AI架构自主地对每起事件进行调查，并在警报射击后几分钟内向Slack发送有条理的汇总。 而不是原始的警报，待命的工程师现在得到可操作的背景，所以他们可以立即行动，而不是花一个小时找出发生了什么。
K圈将平均切入时间缩短至15分钟以下
在本文中，我们描述最初的 Amazon Bedrock Flows 执行架构
B. 支离破碎间接费用的挑战
Circle K的DevOps团队使用Grafana监测多个AWS账户的基础设施。 典型的一天带来提醒，用于容器重启，出错率高，悬空突起，部署失败。 每个警报都是一种有用的信号，但随着系统的扩大，团队想帮助待命工程师迅速看到哪些信号需要注意，有正确的上下文。
当警报发出时，分解涉及一个熟悉的人工步骤序列：审查Amazon Cloud Watch中最近的日志，检查Amazon OpenSearch Service中的应用级别痕迹，检查Amazon弹性容器服务(Amazon ECS)任务状态，并交叉引用GitHub中内部运行本的症状。 这为工程师提供了他们需要的信息，但也表明随着更多的服务投入使用，需要反复收集环境资料。
小组认识到，积极主动的投资可以帮助两个领域。 首先，对每起事件都重复同样的日志查询、健康检查和运行本检查。 第二，分门别类的深度和速度可能因待命工程师熟悉受影响的服务而不同。 随着警示量和服务所有权的增加，每次调查的多种工具之间的切换会减缓反应速度并更难一致地应用小组的最佳做法。
因此，Circle K的DevOps团队发起了一项研究与发展(研发)倡议，以探索AI如何支持监测和事件分类。 目标是使重复性的背景收集自动化，并提供结构化的、可操作的结论，同时在调查期间支持工程师的业务判断。
为什么K圈选择了Amazon Bedrock
小组在建设前评价了几种办法，包括开源 agent自管基础设施框架。 这种做法将意味着拥有模式托管、即时改版、工具协调以及安全控制。
与Amazon Bedrock，球队避免了这起高架起落。 他们可以通过单一的API和互换基础模型访问基础模型而无需重写代码。 因为Circle K已经运行于AWS上，Amazon Bedrock符合了现有的AWS身份和访问管理(IAM)政策，Amazon虚拟私人云(Amazon VPC)配置，以及伐木管。
两项管理能力包括安排和检索。 Amazon 时速 Bedrock帮助了agent管弦乐，赋予了agents本土工具用于调用AWS API，查询数据库，并检索没有自定义管弦乐代码的文件。 Amazon Bedrock Knowledge Bases，全资管理的检索增强生成(RAG)能力，处理内部产品卡的摄取和取回，因此团队不需要单独运行矢量数据库。 这使得DevOps团队专注于即时设计和工具定义。
数据隐私完成图片。 Circle K的事件数据包括操作度量衡，基础设施细节，以及可以包含敏感信息的日志。 Amazon 时速 Bedrock不与模型提供者分享客户数据，也不使用它来训练地基模型。 对于将生产记录和内部服务细节输入系统的团队来说，这种数据处理方法满足了Circle K的安全要求。
解决办法概述
本节介绍团队在Amazon Bedrock Flows上所构建的初始架构。 后一节介绍向Core探员的迁移情况。
该溶液以Amazon Bedrock Flows为管弦乐分层，协调一组专门的agents，各自处理分析的不同部分。 管道在Grafana警报发布Amazon简单通知服务(Amazon SNS)的通知后开始，该通知将其转发至Amazon简单排队服务(Amazon SQS)队列。 AWS Lambda对队列进行投票，提取警戒有效载荷，并引用"流".
在"流"内部，两个agents被平行地引用。 与AWS组织管理的各种工作量账户的AWS资源探索者代理查询基础设施。 它检查 Amazon ECS 任务状态，应用程序负载平衡器目标健康，以及云表的度量和日志。 同时，OpenSearchExplorerAgent查询Amazon OpenSearch Service用于应用级日志和分发痕迹。 此 agent 单独运行，因为 OpenSearch 返回了大量日志数据。 agent 的上下文窗口被隔离。 它还可以独家进入知识库，该知识库内有产品卡来描述每项服务及其依赖性。
在agents两人归还了他们的发现后，GitHubExplorerAgent运行。 它从受影响的服务GitHub寄存器的前两个agents和发现得到输出。 然后，它搜索它们的应用代码，基础设施定义，以及最近承诺确定问题的根源并提出具体的解决方案。 在平行的 agents 之后运行它意味着它完全知道要搜索什么，而不是进行广泛的查询。
所有三个输出都输入了AnalysisPrompt节点，这是一个单一的模型调用，没有附加工具。 它收到AWS资源数据，OpenSearch日志和GitHub上下文，然后产生结构化摘要：可能的根源，支持证据，受影响的服务和所有者，并推荐补救步骤。 即时输出使用一致的格式，待命小组可在2分钟内扫描。
Flow将这个摘要返回到引用了Lambda函数，该函数将其发布到Slack频道。 从警戒到Slack消息的整个过程需要5到15分钟，这取决于事件的复杂性。
下图显示此架构从最初的Grafana警报到最终的Slack摘要，最终结束。
示例分类
A Grafana 提醒：服务上的HTTP 5xx出错率连续3分钟超过5%.
AWS资源ExplorerAgent检查目标账户中的ECS任务健康，发现数据库连接池耗尽，ECS服务尝试重启。 OpenSearchExplorerAgent 返回显示服务“拒绝连接”错误的应用程序日志到其支持的 PostgreSQL 数据库。 它还从知识库取回受影响服务的产品卡，该知识库记录其对数据库的依赖情况。 GitHubExplorerAgent搜索服务库和基础设施文件以及发现 Amazon关系数据库服务(Amazon RDS)定义。 它看到，这项服务打开了每个请求的新的数据库连接，而不是重新使用现有的连接。
分析Prompt得到所有三种结果。 它将时间与预定的营销运动联系起来，该运动在警报前10分钟开始。 它与GitHubExplorerAgent关于建议一个具体编码级固定而不是临时补丁的结论相结合。
最后的Slack消息写道：
缩放到 Amazon 贝德洛克代理 Core
Amazon Bedrock Flows 架构几个月来运作良好，简单事件在不到10分钟内被三分制。 随着时间的推移，每次调查的数据量和背景都有所增长，尽管这套服务保持不变。 有些事件需要较长时间的分析，而单个流程节点则无法完成。
团队首先通过更积极的过滤日志，缩短上下文windows，将某些服务分拆为单独的流程来优化现有设计。 这些步骤有所帮助，但记录量和知识基础继续增加。
Amazon Bedrock Agent Core满足了这一需要。 AgentCore是一个以任何框架或模型在规模上构建，连接，并优化AI agents的平台。 对于这一工作量，关键能力是Agent Core Runtime：完整的调查在一个环节进行，所以每个步骤都可以花上它的数据所需要的时间。 Agent Core Runtime支持长达8小时的会话，远远超出了目前调查的需要。 团队将他们的agent逻辑装入容器并部署在Agent Core Runtime上。
下图显示迁移到Amazon Bedrock Agent Core后的生产架构。
分析逻辑不变，但运行方式已发生变化。 Flows版本的3个专业的agents现在被整合成在Agent Core跑道会话中运行的单个agent. Lambda函数启动会话，而agent执行与前作相同的步骤。 它查询AWS跨账户资源，从Amazon OpenSearch Service取出日志，搜索GitHub，从知识库取出产品卡。 agent在15分钟内将其发现返回到Lambda函数，这在Lambda的最大超时范围内。 Lambda 函数然后将结构化摘要发布到 Slack.
Amazon Bedrock Agent Core Gateway，是Amazon Bedrock Agent Core的一种能力，经纪人agent对其数据源的访问，包括Amazon OpenSearch Service,Amazon Bedrock知识库，和GitHub. 网关将每个源都暴露为agent所引用的工具，它集中了访问控制，使得团队更容易添加或更改数据源。
Agent Core Runtime如何支持球队不断增长的规模
Amazon Bedrock Agent Core Runtime除了更长的会话之外，还增加了三个支持团队不断扩大规模的能力：会话持续，基于OpenTeleometry的可观察性和框架灵活性。
会话持久性：agent在整个调查期间保持中间状态。 如果第一个通过日志会显示一些东西，agent可以运行第二个，更具针对性的通道而不失去它已经建起的上下文。 这符合对复杂、多服务事件进行调查的迭接方式。
可观测性：Agent Core通过Cloud Watch利用OpenTeleometry标准和表面来捕捉每个会话的痕迹。 团队可以看到每个步骤需要多长时间，哪个知识库询问agent制造，以及瓶颈在哪里出现。 这些见解直接注入了即时的调和。
框架灵活性：使用Agent Core Runtime，团队可以部署以多种框架构建的集装箱化agents.
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Artificial Intelligence
How Circle K accelerated DevOps incident triage to under 15 minutes with Amazon Bedrock
This post was co-written with Marek Klatowicz, Cloud DevOps Engineer at Circle K.
Many DevOps teams know the incident triage pattern: An alert fires, and before anyone can fix the problem, an engineer spends 45–60 minutes figuring out what happened. They pull logs, check service health, and correlate symptoms across a handful of tools. All of this happens before remediation can begin.
At Circle K, this challenge scaled with the business. The company operates more than 17,200 convenience and fuel retail stores across 27 countries. Behind the storefronts, a cloud-native infrastructure supports point-of-sale systems, loyalty applications, and electric vehicle (EV) charging applications running on AWS. As the company expanded its digital operations, the triage overhead grew with it. More services meant more alerts, and each one required the same manual investigation before remediation could begin.
To address it, Circle K built an automated triage system using Amazon Bedrock, a fully managed service that makes foundation models (FMs) available through a single API. For model availability by AWS Region, refer to Supported models by AWS Region in Amazon Bedrock. The system uses an agentic AI architecture to investigate each incident autonomously and deliver a structured summary to Slack within minutes of an alert firing. Instead of raw alerts, on-call engineers now receive actionable context, so they can act immediately rather than spending an hour figuring out what happened.
Since deploying this system, Circle K has reduced mean triage time to under 15 minutes, a 67–75 percent reduction based on the team’s internal incident-response metrics.
In this post, we describe the architecture for both the initial Amazon Bedrock Flows implementation and the migration to Amazon Bedrock AgentCore, and share lessons learned from scaling the system.
The challenge of triage overhead
Circle K’s DevOps team monitors infrastructure across multiple AWS accounts using Grafana. A typical day brings alerts for container restarts, elevated error rates, latency spikes, and failed deployments. Each alert is a useful signal, but as the system expanded, the team wanted to help on-call engineers quickly see which signals needed attention, with the right context.
When an alert came in, the triage involved a familiar sequence of manual steps: reviewing recent logs in Amazon CloudWatch, examining application-level traces in Amazon OpenSearch Service, checking Amazon Elastic Container Service (Amazon ECS) task status, and cross-referencing symptoms with internal runbooks in GitHub. This gave engineers the information they needed, but it also showed how much repetitive context gathering would be required as more services were onboarded.
The team recognized two areas where a proactive investment could help. First, the same log queries, health checks, and runbook lookups would be repeated for every incident. Second, the depth and speed of triage could vary with the on-call engineer’s familiarity with the affected service. As alert volume and service ownership grew, switching between multiple tools for each investigation could slow response and make it harder to apply the team’s best practices consistently.
Circle K’s DevOps team therefore started a research and development (R&D) initiative to explore how AI could support monitoring and incident triage. The goal was to automate repetitive context gathering and provide structured, actionable findings, while supporting engineers’ operational judgment during an investigation.
Why Circle K chose Amazon Bedrock
The team evaluated several approaches before building, including open source agent frameworks on self-managed infrastructure. That approach would have meant owning model hosting, prompt versioning, tool orchestration, and security controls.
With Amazon Bedrock, the team avoided that overhead. They could access foundation models through a single API and swap foundation models without rewriting code. Because Circle K already ran on AWS, Amazon Bedrock fit into existing AWS Identity and Access Management (IAM) policies, Amazon Virtual Private Cloud (Amazon VPC) configurations, and logging pipelines.
Two managed capabilities covered orchestration and retrieval. Amazon Bedrock helped with agent orchestration, giving the agents native tool use to call AWS APIs, query databases, and retrieve documents without custom orchestration code. Amazon Bedrock Knowledge Bases, the fully managed Retrieval Augmented Generation (RAG) capability, handled ingestion and retrieval of internal product cards, so the team did not need to run a separate vector database. This let the DevOps team focus on prompt design and tool definitions.
Data privacy completed the picture. Circle K’s incident data includes operational metrics, infrastructure details, and logs that can contain sensitive information. Amazon Bedrock does not share customer data with model providers or use it to train foundation models. For a team feeding production logs and internal service details into the system, this data handling approach met Circle K’s security requirements.
Solution overview
This section describes the initial architecture, which the team built on Amazon Bedrock Flows. A later section covers the migration to AgentCore.
The solution uses Amazon Bedrock Flows as the orchestration layer, coordinating a set of specialized agents that each handle a different part of the analysis. The pipeline starts when a Grafana alert publishes a notification to Amazon Simple Notification Service (Amazon SNS), which forwards it to an Amazon Simple Queue Service (Amazon SQS) queue. An AWS Lambda function polls the queue, extracts the alert payload, and invokes the Flow.
Inside the Flow, two agents are invoked in parallel. The AWSResourceExplorerAgent queries infrastructure across multiple workload accounts managed with AWS Organizations. It checks Amazon ECS task status, Application Load Balancer target health, and CloudWatch metrics and logs. At the same time, the OpenSearchExplorerAgent queries Amazon OpenSearch Service for application-level logs and distributed traces. This agent runs separately because OpenSearch returns large volumes of log data. Isolating it keeps each agent’s context window manageable. It also has exclusive access to the knowledge base, which contains product cards describing each service and its dependencies.
After both agents return their findings, the GitHubExplorerAgent runs. It receives output from the previous two agents and finds the affected service’s GitHub repositories. Then it searches their application code, infrastructure definitions, and recent commits to pinpoint the source of the problem and propose a concrete fix. Running it after the parallel agents means it knows exactly what to search for rather than performing a broad query.
All three outputs feed into an AnalysisPrompt node, which is a single model call with no tools attached. It receives the AWS resource data, the OpenSearch logs, and the GitHub context, then produces a structured summary: probable root cause, supporting evidence, affected service and owner, and recommended remediation steps. The prompt output uses a consistent format that the on-call team can scan in under 2 minutes.
The Flow returns this summary to the invoking Lambda function, which posts it to a Slack channel. The entire process from alert to Slack message takes 5–15 minutes depending on incident complexity.
The following diagram shows this architecture end to end, from the initial Grafana alert to the final Slack summary.
Example triage
A Grafana alert: HTTP 5xx error rate on a service exceeds 5% for 3 consecutive minutes.
The AWSResourceExplorerAgent checks ECS task health in the target account and finds the database connection pool exhausted, with the ECS service attempting restarts. The OpenSearchExplorerAgent returns application logs showing “connection refused” errors from the service to its backing PostgreSQL database. It also retrieves the product card for the affected service from the knowledge base, which documents its dependency on that database. The GitHubExplorerAgent searches the service’s repository and infrastructure files and finds the Amazon Relational Database Service (Amazon RDS) definition. It sees that the service opens a new database connection per request rather than reusing existing ones.
The AnalysisPrompt receives all three results. It correlates the timing with a scheduled marketing campaign that started 10 minutes before the alert. It combines that with the GitHubExplorerAgent’s finding to recommend a concrete code-level fix rather than a temporary patch.
The final Slack message reads:
Scaling to Amazon Bedrock AgentCore
The Amazon Bedrock Flows architecture worked well for months, and simple incidents were triaged in under 10 minutes. Over time, the volume of data and the context behind each investigation grew, even though the set of services stayed the same. Some incidents required longer-running analysis than a single Flow node could complete.
The team first optimized within the existing design by filtering logs more aggressively, shortening context windows, and splitting certain services into separate Flows. These steps helped, but log volumes and the knowledge base kept growing.
Amazon Bedrock AgentCore met this need. AgentCore is a platform to build, connect, and optimize AI agents at scale with any framework or model. For this workload, the key capability was AgentCore Runtime: The full investigation runs in a single session, so each step can take the time its data requires. AgentCore Runtime supports sessions of up to 8 hours, well beyond what the current investigations need. The team containerized their agent logic and deployed it on AgentCore Runtime.
The following diagram shows the production architecture after the move to Amazon Bedrock AgentCore.
The analysis logic stays the same, but the way it runs has changed. The three specialized agents from the Flows version are now consolidated into a single agent running in an AgentCore Runtime session. The Lambda function starts the session, and the agent performs the same steps as before. It queries AWS resources across accounts, pulls logs from Amazon OpenSearch Service, searches GitHub, and retrieves product cards from the knowledge base. The agent returns its findings to the Lambda function within 15 minutes, which is within the maximum Lambda timeout. The Lambda function then posts the structured summary to Slack.
Amazon Bedrock AgentCore Gateway, a capability of Amazon Bedrock AgentCore, brokers the agent’s access to its data sources, including Amazon OpenSearch Service, Amazon Bedrock Knowledge Bases, and GitHub. The Gateway exposes each source as a tool the agent invokes, which centralizes access control and makes it simpler for the team to add or change data sources.
How AgentCore Runtime supported the team’s growing scale
Beyond longer-running sessions, Amazon Bedrock AgentCore Runtime added three capabilities that supported the team’s growing scale: session persistence, OpenTelemetry-based observability, and framework flexibility.
Session persistence: The agent keeps an intermediate state throughout an investigation. If a first pass over the logs surfaces something, the agent can run a second, more targeted pass without losing the context it has already built. This fits the iterative way that complex, multi-service incidents are investigated.
Observability: AgentCore captures traces from each session using OpenTelemetry standards and surfaces through CloudWatch. The team can see how long each step takes, which knowledge base queries the agent makes, and where bottlenecks occur. These insights feed directly into prompt tuning.
Framework flexibility: With AgentCore Runtime, the team can deploy containerized agents built with a wide range of frameworks.
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.