人工智能
与 Crawl4AI 为 Amazon Bedrock 知识库准备网络内容
在构建 Retrieval 增强生成(RAG)应用程序时，回答质量取决于您所摄取的数据。 Amazon Bedrock知识库是Amazon Bedrock的全管理RAG能力。 与 Amazon Bedrock 管理知识 Base，您连接您的数据源，Amazon Bedrock 管理摄入，存储，索引和检索。 您不提供或缩放向量存储，默认情况下嵌入并重新排序使用服务管理模式。
对于网络内容，从Amazon Bedrock Web Crawler数据源连接器开始，用于管理的知识库。 您给出了种子 URL 或站点映射您被授权爬行，它爬行在您设定的范围和速率限制。 计划同步会随着页面变化而保持您的知识库流畅，连接器可以等加载到JavaScript的内容。 您没有建立、主机或操作的爬行器，因此这个连接器是大多数公共网站内容的起点。
一些来源需要更多地控制知识库。 您可能想要只索引每个页面的主要内容区域，删除导航和其它锅炉板，或者在每一个文档中附加您自己的元数据属性。 对于该类源材料，你可以通过用克劳尔4AI(开源 Python crawer)所构建的准备步骤来扩展你的知识库的覆盖范围。 该步骤将每个页面的Markdown和一个元数据侧车写到Amazon简单存储服务(Amazon S3)，知识库通过它的Amazon S3数据源摄取。
在本文中，我们向您演示如何选择合适的网络摄取路径，并对需要额外处理的内容执行自定义准备。 然后，您开始摄入一个管理的知识库，并验证检索质量。 相伴的aws-样本存储器作为样本执行提供了完整的工作流程。
这是分批地摄取已知的网页属性，而不是实时网页在推论时的访问，比如用Amazon Nova来进行Web Grounding.
解决办法概述
从Amazon Bedrock Web Crawler开始，当它能够到达并索引需要的页面时。 它尊重机器人.txt，在第一次完全爬行后，每次同步捕获只添加，更新和删除内容。
使用 Crawl4AI 准备路径，当公共源材料在摄取前需要以下一种或多种：
- 结构化的Markdown准备：在保留标题、列表和表格的同时，使用带有穿透内容过滤器的默认MarkdownGenerator来减少导航和其他锅炉板。 因为挤压是高压的，审查你们网站的代表性产出。
- 受约束、目标明确的提取：只提取可选 CSS 选择器匹配的页面区域。 按种子来源、页面/路径/主机范围、深度和页盖以及URL图案来限制宽度第一。
- 自定义元数据：附加源 URL, craw timettamp，内容类型，以及您自己对每个文档的属性，这样您就可以过滤它们上的检索结果。
由于管理和客户管理的知识库都支持Amazon S3数据源，因此Crawl4AI路径与任一类型一起工作。 这个员额使用一个管理下的知识库和一个客户管理的知识库不同的指出。
此帖余以相伴取样。 将它的架构和默认作为起点，而不是Amazon Bedrock知识库的要求。
默认情况下，样本会在公共子网中运行AWS Fargate任务，通过互联网网关到达您配置的网站。 Crawl4AI的作用在它准备出源文件和元数据旁车后就结束了。 Amazon Bedrock Knowledge Bases从摄入开始，处理剖析、分块、嵌入、索引和检索。 下图显示 Crawl4AI 从调度到检索路径。
图表图中的编号步骤到以下工作流程：
- Amazon EventBridge在配置的时间表上开始Amazon弹性集装箱服务(Amazon ECS)任务。
- 任务爬入您配置的公共网站，并为每个页面准备一个元数据旁卡。
- 任务将准备的文件上传到支持知识库数据源的Amazon S3桶。
- 任务开始Amazon Bedrock知识库摄取工作。
- Amazon Bedrock知识库读取新的和已改变的对象，分析它们，并应用你选择的块策略。
- 它在知识库的矢量储存库中建立嵌入和索引，Amazon Bedrock管理一个管理的知识库。
- 您的基因AI应用询问相关块的知识库或者通过引用生成的答复。
任务从启动时的AWS系统管理器参数存储器读取其非秘密管道配置。
工作流程只处理公共页面，而不执行经认证的会议。 对于需要登录的站点，首先评价Amazon Bedrock Web Crawler认证选项，这些选项包括基本，基于形式，以及SAML认证。 以认证方式扩展样本需要会话处理，AWS保密管理器中的证书，以及安全审查。
工作流程如何运作
AWS Cloud Development Kit(AWS CDK)应用软件在部署时将执行内容以容器图像来包装。 时刻表起火后，容器会运行脚本/容器 intrypoint.py并按顺序执行爬行，上传和摄取。 以下三个步骤解释了核心逻辑。
第1步：爬行并提取内容
爬行步骤执行被绑定的爬行，并为每个成功准备的页面写出一个有元数据侧车的Markdown文件。 以下摘录被简化。 模块 src/crawl4ai bedrock kb/crawl.py包含完整的种子间隔，URL过滤和范围界定，页面限制，并重定向检查：
Amazon Bedrock Knowledge Bases支持Amazon S3数据源的几种文档格式，包括纯文本，HTML,Markdown,PDF，和DOCX. 工作流程写入了Markdown，因为它保留了标题，列表，和表格，而没有携带所有原始的HTML来摄取。
第2步：上传到 Amazon S3
上传步骤会在工作流程的 s3Prefix 下，将每个已准备好的Markdown 文档及其匹配的元数据侧车发送到配置的 S3 桶。 对于page.md，侧车必须被命名为page.md.metadata.json，完整的源文件名有.metadata.json附件。 工作流程将每台侧车存放在与其文档相同的S3位置。 Amazon Bedrock将每个元数据侧车限制在10 KB. 如果一辆侧车超过这个限制，运行会在其写出任何文件前出现出错而停止。
副车厢中的每个属性都使用管理及客户管理的知识库接受Amazon S3数据源的打出格式。 以下简称侧车，格式化为可读性，显示 3 由工作流程所撰写的属性：
管道只上传由当前爬行生成的路径，因此不发送已过时的本地文件。
第3步：让Amazon Bedrock知识库块、嵌入和索引
内容在S3之后，一个摄取工作将每个文件交给数据源的摄取管道。 默认情况下，一个管理下的知识库会用它管理下的解析器来解析每个文件，它会自动为您的内容选择解析方法。 在连接数据源时选择块策略。 在创建生产数据源之前，您不能修改它，所以在您创建生产数据源之前要先评估选项。 管理下的知识库提供以下零星战略：
- 默认块使用300个符号的固定大小块，重叠20%。
- 固定大小的块分割内容，以您设定的最大符号计数和重叠百分比。
- 对于您已经拆分的内容，没有将每个文件当作一个块处理。
春克边界和检索结果取决于内容和所选策略，因此将选项与代表性页面和评价问题进行比较。 为了日后改变策略，创建一个新的数据源。 更多信息见自定义摄入数据源。 由客户管理的知识库还提供语义和分级划分以及额外的解析选择。 详情见内容如何对知识库进行分解。
然后工作流程开始摄取工作，因此知识库处理新的文件。 以下从src/crawl4ai bedrock kb/ingest.py 显示所调取的简化节录：
同样的"StartIntoptionJob"和"GetIntoptionJob"为两种知识库类型工作，也可以在控制台上监控进度。
部署解决方案
在这个区域中，您将工作流程配置到您的 AWS 账户中，使用 AWS CDK. 只有知识库的先决条件在两种知识库类型之间有所不同。 当您完成后，请遵循清理部分的步骤来移除资源。
先决条件
在开始前，确保你有：
- AWS账户。
- 下列知识库之一：
- 该员额在AWS区域使用管理下的知识库，那里有管理下的知识库。 关于指令，参见"创建管理的知识库"和"Amazon S3"连接器。 在数据源上，留下元数据文件前缀为空，因为工作流程会将每个元数据旁卡与文档并列。
- 现有客户管理的知识库。 要添加 Amazon S3 数据源，请参见 Connect to Amazon S3 您的知识库。 工作流程为每个文档写一个元数据旁卡，以确认您的矢量存储可以存储并过滤这些属性，如元数据和过滤中描述的。 例如，Amazon OpenSearch 服务器无向量索引必须使用faiss引擎。 Amazon Aurora矢量指数需要自定义元数据列或每个元数据属性的一列。 通过 Amazon S3 矢量器，可以使用 Amazon S3 的能力，您可以在每个矢量上附加最多1个自定义元数据KB，所以要保持您添加的属性短。
- 该知识库的Amazon S3数据来源，其状况为AVAILable。 它可以是新的数据源，也可以是您已经使用的数据源。 它必须使用您为此工作流程配置的 S3 桶。 如果数据源通过前缀或文件模式过滤内容，过滤器必须允许工作流程在其 s3Prefix (default web-content/)下所写入的文件。
- 知识库，S3水桶，还有你部署的堆积物
- 授权浏览您配置的公共网站。
- 一个部署工作站，配有Git,Python 3.10或后期，Node.js,AWS指挥线接口(AWS CLI) 第2版，AWS CDK CLI v2，以及运行中的Docker兼容容器引擎。 部署指南列出了最小的AWS CDK CLI版本。
- AWS证书配置在部署工作站上，允许拖曳目标环境并部署CDK堆。
- 目标账户和地区中一个有靴子的CDK环境。 部署步骤包括靴子陷阱命令。
你只需要工作站来建造和部署部署后，调度在AWS运行任务。
部署预定的工作流程
清除同伴寄存器并安装部署依赖性：
保持Docker运行。 将 AWS DEFAULT Region 设定为您的知识库区域，因为 cdk 部署使用来自您的环境或 AWS 配置文件，而不是靴子上的命令。 必要时，将目标账户和区域绑一次。 然后用您的知识库、数据源、 S3 桶和授权种子 URL 来部署堆栈：
替换每个示例值，包括 AcoUNT ID，区域，以及所需的四个上下文密钥，以您自己的。 示例值在策略上是有效的，因此一个未经编辑的命令会部署一个堆栈，以指向您没有的桶。 也可以将这些密钥设置在cdk.json中。
最大深度和最大Pages值将第一次爬行限制在一个链接级别和25页。 在您审查已准备好的输出后，提高它们。 要预览哪一页
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Artificial Intelligence
Prepare web content for Amazon Bedrock Knowledge Bases with Crawl4AI
When you build Retrieval Augmented Generation (RAG) applications, answer quality depends on the data that you ingest. Amazon Bedrock Knowledge Bases is the fully managed RAG capability of Amazon Bedrock. With Amazon Bedrock Managed Knowledge Base, you connect your data sources, and Amazon Bedrock manages ingestion, storage, indexing, and retrieval for you. You don’t provision or scale a vector store, and embedding and reranking use service-managed models by default.
For web content, start with the Amazon Bedrock Web Crawler data source connector for managed knowledge bases. You give it the seed URLs or sitemaps that you’re authorized to crawl, and it crawls within the scope and rate limits that you set. Scheduled syncs keep your knowledge base current as pages change, and the connector can wait for content that loads with JavaScript. There’s no crawler for you to build, host, or operate, so this connector is the starting point for most public website content.
Some sources need more control over what reaches the knowledge base. You might want to index only the main content region of each page, remove navigation and other boilerplate, or attach your own metadata attributes to every document. For that class of source material, you can extend the reach of your knowledge base with a preparation step built with Crawl4AI, an open source Python crawler. The step writes Markdown and a metadata sidecar for each page to Amazon Simple Storage Service (Amazon S3), and the knowledge base ingests both through its Amazon S3 data source.
In this post, we show you how to choose the appropriate web ingestion path and implement custom preparation for content that needs additional processing. You then start ingestion into a managed knowledge base and validate retrieval quality. The companion aws-samples repository provides the complete workflow as a sample implementation.
This is batch ingestion of known web properties, not real-time web access at inference time such as Web Grounding with Amazon Nova.
Solution overview
Start with the Amazon Bedrock Web Crawler when it can reach and index the pages you need. It respects robots.txt, and after the first full crawl, each sync captures only added, updated, and deleted content.
Use the Crawl4AI preparation path when public source material needs one or more of the following before ingestion:
- Structured Markdown preparation: Use DefaultMarkdownGenerator with a pruning content filter to reduce navigation and other boilerplate while preserving headings, lists, and tables. Because pruning is heuristic, review representative output from your sites.
- Bounded, targeted extraction: Extract only the page region that an optional CSS selector matches. Constrain breadth-first traversal by seed origin, page/path/host scope, depth and page caps, and URL patterns.
- Custom metadata: Attach source URLs, crawl timestamps, content type, and your own attributes to each document, so that you can filter retrieval results on them.
Because managed and customer-managed knowledge bases both support Amazon S3 data sources, the Crawl4AI path works with either type. This post uses a managed knowledge base and notes where a customer-managed knowledge base differs.
The rest of this post follows the companion sample. Treat its architecture and defaults as a starting point, not as requirements of Amazon Bedrock Knowledge Bases.
By default, the sample runs the AWS Fargate task in a public subnet, where it reaches your configured websites through an internet gateway. Crawl4AI’s role ends once it has prepared the source documents and metadata sidecars. From ingestion onward, Amazon Bedrock Knowledge Bases handles parsing, chunking, embedding, indexing, and retrieval. The following diagram shows the Crawl4AI path from the schedule through to retrieval.
The numbered steps in the diagram map to the following workflow:
- Amazon EventBridge starts the Amazon Elastic Container Service (Amazon ECS) task on AWS Fargate on the configured schedule.
- The task crawls the public websites you configured and prepares Markdown with a metadata sidecar for each page.
- The task uploads the prepared files to the Amazon S3 bucket that backs the knowledge base data source.
- The task starts an Amazon Bedrock Knowledge Bases ingestion job.
- Amazon Bedrock Knowledge Bases reads the new and changed objects, parses them, and applies the chunking strategy you selected.
- It creates embeddings and indexes them in the knowledge base’s vector store, which Amazon Bedrock manages for a managed knowledge base.
- Your generative AI application queries the knowledge base for relevant chunks or for a generated response with citations.
The task reads its non-secret pipeline configuration from AWS Systems Manager Parameter Store at startup.
The workflow processes public pages only and doesn’t implement authenticated sessions. For sites that require sign-in, evaluate the Amazon Bedrock Web Crawler authentication options first, which include basic, form-based, and SAML authentication. Extending the sample with authentication requires session handling, credentials in AWS Secrets Manager, and a security review.
How the workflow works
The AWS Cloud Development Kit (AWS CDK) app packages the implementation in a container image during deployment. When the schedule fires, the container runs scripts/container_entrypoint.py and executes crawl, upload, and ingestion in sequence. The following three steps explain the core logic.
Step 1: Crawl and extract content
The crawl step performs bounded crawling and writes a Markdown file with a metadata sidecar for each successfully prepared page. The following excerpt is simplified. The module src/crawl4ai_bedrock_kb/crawl.py contains the complete seed pacing, URL filtering and scoping, page limits, and redirect checks:
Amazon Bedrock Knowledge Bases supports several document formats for Amazon S3 data sources, including plain text, HTML, Markdown, PDF, and DOCX. The workflow writes Markdown because it preserves headings, lists, and tables without carrying all raw HTML into ingestion.
Step 2: Upload to Amazon S3
The upload step sends each prepared Markdown document and its matching metadata sidecar to the configured S3 bucket, under the workflow’s s3Prefix. For page.md, the sidecar must be named page.md.metadata.json, the complete source filename with .metadata.json appended. The workflow stores each sidecar in the same S3 location as its document. Amazon Bedrock limits each metadata sidecar to 10 KB. If a sidecar would exceed that limit, the run stops with an error before it writes either file.
Each attribute in the sidecar uses the typed format that both managed and customer-managed knowledge bases accept for Amazon S3 data sources. The following abbreviated sidecar, formatted for readability, shows three of the attributes that the workflow writes:
The pipeline uploads only the paths produced by the current crawl, so stale local files are not sent.
Step 3: Let Amazon Bedrock Knowledge Bases chunk, embed, and index
After content is in S3, an ingestion job hands each document to the data source’s ingestion pipeline. By default, a managed knowledge base parses each file with its managed parser, which automatically selects a parsing approach for your content. You select the chunking strategy when you connect the data source. You can’t change it afterward, so evaluate the options before you create a production data source. A managed knowledge base offers the following chunking strategies:
- Default chunking uses fixed-size chunks of 300 tokens with 20 percent overlap.
- Fixed-size chunking splits content by a maximum token count and an overlap percentage that you set.
- No chunking treats each file as a single chunk, for content that you have already split.
Chunk boundaries and retrieval results depend on the content and the selected strategy, so compare the options with representative pages and evaluation questions. To change strategies later, create a new data source. For more information, see Customize ingestion for a data source. A customer-managed knowledge base also offers semantic and hierarchical chunking and additional parsing options. For details, see How content chunking works for knowledge bases.
The workflow then starts an ingestion job so the knowledge base processes the new documents. The following simplified excerpt from src/crawl4ai_bedrock_kb/ingest.py shows the call:
The same StartIngestionJob and GetIngestionJob calls work for both knowledge base types, and you can also monitor progress on the console.
Deploy the solution
In this section, you deploy the workflow into your AWS account with AWS CDK. Only the knowledge base prerequisites differ between the two knowledge base types. When you’re finished, follow the steps in the Clean up section to remove the resources.
Prerequisites
Before you start, make sure that you have:
- An AWS account.
- One of the following knowledge bases:
- A managed knowledge base, which this post uses, in an AWS Region where managed knowledge bases are available. For instructions, see Create a managed knowledge base and the Amazon S3 connector. On the data source, leave Metadata files prefix empty, because the workflow writes each metadata sidecar alongside its document.
- An existing customer-managed knowledge base. To add an Amazon S3 data source, see Connect to Amazon S3 for your knowledge base. The workflow writes a metadata sidecar for every document, so confirm that your vector store can store and filter those attributes, as described in Metadata and filtering. For example, an Amazon OpenSearch Serverless vector index must use the faiss engine. An Amazon Aurora vector index needs a custom metadata column or a column for each metadata attribute. With Amazon S3 Vectors, a capability of Amazon S3, you can attach up to 1 KB of custom metadata to each vector, so keep attributes that you add short.
- An Amazon S3 data source in that knowledge base whose status is AVAILABLE. It can be a new data source or one you already use. It must use the S3 bucket that you configure for this workflow. If the data source filters content by prefix or file pattern, the filters must allow the files that the workflow writes under its s3Prefix (default web-content/).
- The knowledge base, the S3 bucket, and the stack you deploy all in one Region.
- Authorization to crawl the public websites you configure.
- A deployment workstation with Git, Python 3.10 or later, Node.js, the AWS Command Line Interface (AWS CLI) version 2, the AWS CDK CLI v2, and a running Docker-compatible container engine. The deployment guide lists the minimum AWS CDK CLI version.
- AWS credentials configured on the deployment workstation with permission to bootstrap the target environment and deploy the CDK stack.
- A bootstrapped CDK environment in the target account and Region. The deployment steps include the bootstrap command.
You need the workstation only to build and deploy. After deployment, the schedule runs the task in AWS.
Deploy the scheduled workflow
Clone the companion repository and install the deployment dependencies:
Keep Docker running. Set AWS_DEFAULT_REGION to your knowledge base’s Region first, because cdk deploy uses the Region from your environment or AWS profile, not the one in the bootstrap command. Bootstrap the target account and Region once, if needed. Then deploy the stack with your knowledge base, data source, S3 bucket, and authorized seed URLs:
Replace every example value, including ACCOUNT_ID, REGION, and the four required context keys, with your own. The example values are syntactically valid, so an unedited command deploys a stack that points at a bucket you don’t own. You can also set these keys in cdk.json.
The maxDepth and maxPages values limit the first crawl to one link level and 25 pages. Raise them after you review the prepared output. To preview which pages
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.