- 最近对Meta隐私意识基础设施(PAI)的深度潜水
- 在数据通过Meta的不同数据和处理系统移动时，必须坚持隐私要求。 从历史上看，支持一项新的要求意味着在每个系统中分别执行同样的执行逻辑。 PrivacyLib用一个运行在Meta的数据和处理系统的客户端内部的普通库来取而代之。
- 系统与PrivacyLib一次集成，新的隐私能力插入，作为重新使用这种集成的模块。 隐私Lib每秒进行数十亿次检查，每个模块模拟并推出变化。
- 对于内部的AI agents，其数据访问路径可能无法事先知道，工具调用被支持的路径将agent上下文带入隐私Lib. 所传播的上下文将每个访问连接到它的会话，因此模块可以观察数据使用并屏蔽不允许的操作。
在Meta，许多隐私要求最终得出同样的问题：我们能证明一个数据只在允许的地方进行吗？在Meta的规模上，我们不只依靠人工的隐私审计，它可以随着系统的变化而脱离同步，而是开发了隐私意识基础设施(PAI)，以在基础设施内部回答这个问题。 PAI提供扎入Meta基础设施的高效而可靠的一等隐私构造，使产品能够快速地发展。
在前几篇文章中，我们谈到了数据理解、数据序列和政策区(我们的信息流控制技术)如何在PAI中发挥作用。 现在，我们正在潜入PrivacyLib，一个普通的库，它运行在客户数百个我们的数据和处理系统在数据访问点。
PrivacyLib在更广泛的PAI工作流程的最后阶段发挥重要作用：持续地执行和监测数据流动。
在PrivacyLib之前，我们将隐私执法单独整合到每个数据的客户端和每个要求的处理系统。 通过这种办法，跨系统执行一项新的要求可能需要几个月。 每项新的要求，无论是目的限制还是数据流动的发现，都需要相同的一套能力：涵盖广泛的系统；在数周内而不是数月内部署；在不成为暂时性问题的情况下运行每个数据访问；在不减少机队的情况下推出执行变化。 随着时间推移，随着要求和系统的演变，这种办法更难以扩大规模。
PrivacyLib将系统集成与要求的执行相分离。 我们把每个系统都融为一体当在数据访问点可以满足一项要求时，我们将其作为一个共享于已经集成的系统的隐私Lib模块加以实施，因此，新的要求不再需要在每个系统中单独实施。
模块有两种：
- 执行模块可以阻止数据访问。 例如，政策区执行目的限制。
- 观察模块只记录数据访问。 例如，隐私探测器收集用于发现数据流的运行时信号。
随着内部的AI agents成为了获取数据的新方法，我们还加入了Agent可观察性模块来观测与agent相关的数据访问事件并帮助保护agents能够到达的数据。 已经与PrivacyLib集成的系统在没有新的集成工作的情况下捡起它，这就是收购的集成后：每个新的隐私能力作为模块而来，而不是作为一个项目。
Meta规模的工程挑战和隐私方法
隐私Lib在 Meta 运行于广泛的数据访问路径：
- 一百多家图书馆，用于获取数据服务，从钥匙/价值商店到AI功能。
- 提供数千个已部署图书馆版本的制作服务。
- 每天有数百万资产被查取。
- 每秒有十亿张支票
如此规模的运行造成了三个工程挑战：
走过：保护信件元数据
为了说明一切如何在 Meta 的规模上走到一起，让我们走过一个隐私要求
用户通过诸如Messor等支持端到端加密的应用程序来交换消息。 邮递员根据适用的法律，在传递这些消息时，其后端系统处理非内容信息数据(即信息元数据)，如信息发送或发送时的信息，以提高产品性能，发现滥用行为并保护用户免受有害行为。
各种各样的后端服务使用在服务过程中运行的客户端库"Logger"来将这个元数据写入数据仓表格。 在写入记录之前，Logger内部的PrivativeLib集成会引用一个执行模块"政策区"(Policy Zones)，以检查元数据是否会流向目的表。 这一检查分为两部分：需要哪些投入才能作出决定，以及检查是如何在运行时间得到实际评价的。 在研究了这两个部分之后，我们审视了隐私利卜如何在系统之间进行规模划分，以及如何在不干扰路径中服务的情况下展开变化。
注释信件元数据
每个 Logger 目的地表都有一个 LoggerConfig 来描述写给它的记录。 元数据Trace LocgerConfig 定义了用于信件发送痕迹的这种配置，并将其标记为包含信件元数据：
最终类元数据跟踪LoggerConfig 扩展 LoggerConfigBase {
(一) 使用MessagingTrafficMetadata PolicyTrait;
公共函数配置( I)
Logger 域定义 $fd
: Logger 字段定义器{
$- > 在表级政策中注明；
/ 外地和目的地定义
}
}
一个伐木目的地可以有很多写法，所以描述其输出数据的注释属于目的地，而不是每个写给它的服务。 如果每个服务单位分别宣布注释，声明可能会有分歧，或者新作者可以完全省略。 在"LoggerConfig"中声明一次，使每个编剧对数据都有相同的描述。
对于此目的地，MessagingTrafficMetadata PolicyTrait 提供了 MESSAGING TRAFFIC METADATA 表级注释，并在目标上附上了Annotate with TableLevel Policy ()。
LoggerConfig是Logger自己的计划，每个系统都用自己的格式描述自己的数据。 要给PrivacyLib一个单一的消费形式，Logger的代码生成管道将配置转换为PrivacyLib相容的文物。 生成的元数据TraceLoggerConfigPrivacyLibArtifact来源于元数据TraceLoggerConfig，并带有目的地身份和消息注释：
/**
* @ codegen- source 元数据跟踪
*/
类元数据跟踪
扩展 Logger InputPrivacy LibArtiffact { 允许使用
最终的公用静态函数 get AssetIDImpl ()
): LoggerInputAssetID {
返回 Logger 输入定义ID:: get ()
“ metadata Trace 标记符”
);
}
被保护的静态函数获得目的政策变量( O)
：主题政策变量{
返回目的政策：
主题政策Name:MESSAGING TRAFFIC METATATA
);
}
/ 省略了其他生成的字段。
}
上面生成的类被Logger的客户端Hack,Meta所打入的PHP语言所使用。 对于其C++和Python客户端，Logger发布目的地身份和对中央目录的说明。 PrivacyLib利用这些信息生成语言特有文物。 在下一节的 C++ 示例中，PrivacyLib 使用 LoggerConfig 名称为目的地加载 C++ 文物。
运行检查
前一节的文物描述了一个Logger目的地。 隐私 Lib还需要知道Logger写作的意义和应该运行的模块。 Logger的Hack整合定义
$config - > 图书馆 Config () - > 可操作的 PrivacyLib ()
隐私操作：write ()
- >输入( I)
隐私输入：输入( I)
名称 ILogger 输入PrivaceyLibPayload 提供程序
),
)
- >执行(
隐私列表执行：区 ()
区数据类型：LOGGER
)
- > 附有说明()
- >启用()
隐私写作执行：：遥测日志( ) - > 启用( )
// 其他模块
),
);
PrivacyLib Operation::write () 为 Logger 声明 PrivacyLib 的写操作，执行() 列出运行在它上的模块。 政策区模块可影响结果，而遥测日志模块记录评价而不影响评价。 在此添加一个模块不需要更改单个目的地或Logger呼叫站点。 ZoneDataType:::LOGGER告诉政策区它正在检查Logger目的地。
在 C++ 服务中，Logger 构造隐私Lib 请求如下：
Const 汽车资产列表 =
std::vector<std::string>{configName};
隐私LibBuilder 构建器( I)
"洛杉矶作家"
资产列表
LoggerWrite APIenum 帮助器：: apiToHashedString ()
政策强制LoggerWriteAPI:::log)
打电话者：LOGGER
获取 LoggerWrite 执行 ();
自动结果 =
隐私请求(std:move (buildinger)). 发送(;
隐私Lib 使用配置Name 来装入目的地的文物。 散列政策强制LoggerWriteAPI:log值映射Logger的API到隐私Lib的写入操作。 剩下的构建者参数确定Logger为调用者，并提供执行写入的操作。 发送()引用所配置的模块并记录其结果。 对于政策区模块，PrivativeLib提供了来自所生成的文物和当前处理背景的目的地说明。 例如，政策区评价两个条件：
- 当前上下文是允许使用消息元数据。
- 目的地文物带有MESSAGING TRAFFIC MEDATA注释。
启用执行时，如果任一条件都失败，政策区模块将拒绝写入。 隐私 Lib 将执行模块的结果合并为一个响应。 拒绝在洛格向下游发送记录之前停止了写作。 观测模块发射遥测而不影响反应。
跨系统扩展隐私检查
隐私Lib使隐私检查能够通过一个共同的执行模式、轻量级评价和遥测以及覆盖绕过综合客户的请求，在各种系统中扩大范围。
系统通用执行模式
隐私 Lib的集成跨越了非常不同的API的系统：密钥价值商店，图表数据库，ML管线，文件系统，和Logger等记录系统。 每个集成图显示其访问点为五个共同操作中的一个，PrivacyLib引用其模块：
- 读取：在获取数据时运行。
- 写：存储数据时运行。
- RPC：呼叫另一服务时运行。
- 请求条目：在服务开始处理请求时运行一次。
- 内置执行：将一个特定的计算包在一个更大的请求中，例如评价一个特性。 在计算过程中获得的数据使用其内部隐私背景进行评估。
模块是针对这五个操作，而不是针对每个系统的本地API。 读取，写取，RPC也有操作后变体，因此观察模块可以在访问决定后运行。 一旦一个系统提供了映射，每个相容的模块都可以再利用它，这就是为什么允许在已经集成的系统上添加一个模块只需要几天的时间。
隐私Lib运行在数据和处理系统的客户端内部，因为客户端已经具备了模块所需的目的身份和请求上下文。 将评价维持在同一过程中，可避免在每次访问时将这些意见跨出过程界限。 这种设计也有两个优点：
- 更丰富的语境：隐私Lib可以使用打电话者的身份、终点和目的，而这些信息仍然可用。 如果在数据存储处对检查进行评价，则经常丢失这些信息。
- 失败隔离：运行该版本代码的服务存在错误变化，将共享的数据存储保存在失败边界之外。
保持计算成本的检查
进行中评价
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
- This latest deep dive into Meta’s Privacy Aware Infrastructure (PAI) takes a look at PrivacyLib.
- A privacy requirement must hold as data moves through Meta’s diverse data and processing systems. Historically, supporting a new requirement meant implementing the same enforcement logic separately in each system. PrivacyLib replaces this with a common library that runs inside the clients of Meta’s data and processing systems.
- A system integrates with PrivacyLib once, and new privacy capabilities plug in as modules that reuse that integration. PrivacyLib performs billions of checks every second, and changes are simulated and rolled out per module.
- For internal AI agents, whose data access paths may not be known in advance, tool calls on supported paths carry agent context to PrivacyLib. The propagated context links each access to its session, so modules can observe data use and block operations that are not allowed.
At Meta, many privacy requirements ultimately come down to the same question: Can we prove that a piece of data goes only where it is allowed? At Meta’s scale, rather than relying solely on manual privacy audits, which can go out of sync as systems change, we have developed Privacy Aware Infrastructure (PAI) to answer this question within the infrastructure itself. PAI offers efficient and reliable first-class privacy constructs embedded in Meta’s infrastructure so that products can keep evolving quickly.
In previous posts, we have covered how data understanding, data lineage, and Policy Zones (our information flow control technology) play a role in PAI. Now, we are diving deep into PrivacyLib, a common library that runs inside the clients of hundreds of our data and processing systems at the point of data access.
PrivacyLib plays an important role in the final stage of the broader PAI workflow: continuously enforcing and monitoring data flows.
Before PrivacyLib, we integrated privacy enforcement separately into the clients of each data and processing system for each requirement. With this approach, implementing a new requirement across systems could take months. Each new requirement, whether purpose limitation or data flow discovery, needed the same set of capabilities: covering a wide range of systems; deploying in weeks rather than months; running on every data access without becoming a latency problem; and rolling out enforcement changes without taking the fleet down. Over time, this approach became harder to scale as requirements and systems evolved.
PrivacyLib separates system integration from requirement implementation. We integrate each system once. When a requirement can be met at the point of data access, we implement it as a PrivacyLib module shared across the systems already integrated, so a new requirement no longer needs a separate implementation in each system.
There are two kinds of modules:
- Enforcing modules can block a data access. For example, Policy Zones enforces purpose limitation.
- Observing modules only record a data access. For example, Privacy Probes collects the runtime signals used to discover data flows.
As internal AI agents became a new way of accessing data, we also added the Agent Observability module to observe agent-related data access events and help safeguard the data that agents can reach. Systems already integrated with PrivacyLib picked it up without new integration work, which is what integrating once buys: each new privacy capability arrives as a module, not as a project.
Engineering Challenges and Approaches of PrivacyLib at Meta’s Scale
PrivacyLib operates across a broad range of data access paths at Meta:
- More than a hundred libraries used to access data services, from key/value stores to AI features.
- Thousands of production services with versions of the deployed library.
- Millions of assets being accessed daily.
- Billions of checks every second.
Operating at this scale creates three engineering challenges:
Walkthrough: Protecting Messaging Metadata
To illustrate how everything comes together at Meta’s scale, let’s walk through a privacy requirement that needs to be upheld when processing messaging metadata.
Users exchange messages through apps like Messenger, which supports end-to-end encryption. As Messenger routes and delivers those messages, subject to applicable law, its backend systems process non-content messaging data (i.e., messaging metadata), such as information about when messages are sent or delivered, to improve product performance, detect abuse, and protect users from harmful conduct.
A variety of these backend services use Logger, a client library that runs in the service process, to write this metadata to data warehouse tables. Before writing a record, the PrivacyLib integration inside Logger invokes Policy Zones, an enforcing module, to check whether the metadata may flow to the destination table. This check is broken down into two parts: what inputs need to be available to make a decision, and how the check is actually evaluated at runtime. After examining these two parts, we look at how PrivacyLib scales across systems and how changes roll out without disrupting the services in the path.
Annotating Messaging Metadata
Every Logger destination table has a LoggerConfig that describes the records written to it. MetadataTraceLoggerConfig defines one such configuration for message-delivery traces and marks it as containing messaging metadata:
final class MetadataTraceLoggerConfig extends LoggerConfigBase {
use MessagingTrafficMetadataPolicyTrait;
public function configure(
LoggerFieldDefiner $fd,
): LoggerFieldDefiner {
$this->annotateWithTableLevelPolicy();
// Field and destination definitions
}
}
A logging destination can have many writers, so the annotation describing its output data belongs to the destination rather than to each service that writes to it. If every service declared the annotation separately, the declarations could disagree, or a new writer could omit it entirely. Declaring it once in LoggerConfig gives every writer the same description of the data.
For this destination, MessagingTrafficMetadataPolicyTrait supplies the MESSAGING_TRAFFIC_METADATA table-level annotation, and annotateWithTableLevelPolicy() attaches it to the destination.
LoggerConfig is Logger’s own schema, and each system describes its data in its own format. To give PrivacyLib a single form to consume, Logger’s code-generation pipeline translates the configuration into a PrivacyLib-compatible artifact. The generated MetadataTraceLoggerConfigPrivacyLibArtifact is derived from MetadataTraceLoggerConfig and carries the destination identity and the messaging annotation:
/**
* @codegen-source MetadataTraceLoggerConfig
*/
class MetadataTraceLoggerConfigPrivacyLibArtifact
extends LoggerInputPrivacyLibArtifact {
final public static function getAssetIDImpl(
): LoggerInputAssetID {
return LoggerInputAssetID::get(
'MetadataTraceLoggerConfig',
);
}
protected static function getPurposePolicyVars(
): TPurposePolicyVars {
return PurposePolicy::makeVars(
TPurposePolicyName::MESSAGING_TRAFFIC_METADATA,
);
}
// Other generated fields omitted.
}
The generated class above is used by Logger’s client in Hack, Meta’s typed PHP language. For its C++ and Python clients, Logger publishes destination identities and annotations to a central catalog. PrivacyLib uses that information to generate language-specific artifacts. In the C++ example in the next section, PrivacyLib uses the LoggerConfig name to load the C++ artifact for the destination.
Running the Check
The artifact from the previous section describes one Logger destination. PrivacyLib also needs to know what a Logger write means and which modules should run. Logger’s Hack integration defines that once for all destinations:
$config->libraryConfig()->enablePrivacyLib(
PrivacyLibOperation::write()
->input(
PrivacyLibInput::instance(
nameof ILoggerInputPrivacyLibPayloadProvider,
),
)
->enforcements(
PrivacyLibWriteEnforcement::zones(
ZoneDataType::LOGGER,
)
->withPropagation()
->enabled(),
PrivacyLibWriteEnforcement::telemetryLogging()->enabled(),
// Other modules
),
);
PrivacyLibOperation::write() declares PrivacyLib’s write operation for Logger, and enforcements() lists the modules that run on it. The Policy Zones module can affect the result, while the Telemetry Logging module records the evaluation without affecting it. Adding a module here requires no changes to individual destinations or Logger call sites. ZoneDataType::LOGGER tells Policy Zones that it is checking a Logger destination.
In C++ services, Logger constructs the PrivacyLib request as follows:
const auto assetList =
std::vector<std::string>{configName};
PrivacyLibBuilder builder(
"LOGGER_WRITE_CLIENT",
assetList,
LoggerWriteAPIEnumHelper::apiToHashedString(
PolicyEnforcedLoggerWriteAPI::log),
Caller::LOGGER,
getLoggerWriteImplementation());
auto result =
PrivacyLibRequest(std::move(builder)).dispatch();
PrivacyLib uses configName to load the destination’s artifact. The hashed PolicyEnforcedLoggerWriteAPI::log value maps Logger’s API to PrivacyLib’s write operation. The remaining builder arguments identify Logger as the caller and provide the implementation that performs the write. dispatch() invokes the configured modules and records their outcomes. For the Policy Zones module, PrivacyLib provides the destination annotations from the generated artifact and the current processing context. For this example, Policy Zones evaluates two conditions:
- The current context represents an allowed use of messaging metadata.
- The destination artifact carries the MESSAGING_TRAFFIC_METADATA annotation.
When enforcement is enabled, the Policy Zones module denies the write if either condition fails. PrivacyLib combines the results from enforcing modules into one response. A denial stops the write before Logger sends the record downstream. Observing modules emit telemetry without affecting the response.
Scaling Privacy Checks Across Systems
PrivacyLib enables privacy checks to scale across heterogeneous systems through a common execution model, lightweight evaluation and telemetry, and coverage for requests that bypass integrated clients.
A Common Execution Model Across Systems
PrivacyLib’s integrations span systems with very different APIs: key-value stores, graph databases, ML pipelines, filesystems, and logging systems like Logger. Each integration maps its access points to one of five common operations, and PrivacyLib invokes its modules at each:
- Read: runs when fetching data.
- Write: runs when storing data.
- RPC: runs when calling another service.
- Request entry: runs once when a service begins handling a request.
- Inline execution: wraps a specific computation inside a larger request, such as evaluating a feature. Data accesses made during that computation are evaluated using its inline privacy context.
Modules are written against these five operations rather than against each system’s native API. Reads, writes, and RPCs also have post-operation variants, so observing modules can run after the access decision has been made. Once a system provides the mapping, every compatible module can reuse it, which is why enabling an additional module on an already-integrated system takes only days.
PrivacyLib runs inside the clients of data and processing systems because the clients already have the destination identity and request context that modules need. Keeping the evaluation in the same process avoids carrying those inputs across process boundaries on every access. This design also has two advantages:
- Richer context: PrivacyLib can use the caller’s identity, endpoint, and purpose while that information is still available. This information is frequently lost if the check is evaluated on the datastore.
- Failure isolation: A faulty change is contained to the service running that version of the code, keeping the shared datastore outside the failure boundary.
Keeping Checks Computationally Inexpensive
In-process evaluati
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.