去年，我们推出 GEM，该行业最大的推荐系统(RecSys)基础模型。 GEM建立了Meta的第一个LLM启发的广告推荐范式，驱动了主要的性能收益。 在这个基础上，我们的下一个目标是用下一代硬件进一步扩大计算和智能传输，并将这种能力扩展到Meta的整个广告生态系统。 这跨越数十个表面和目标，在认识、考虑和转换方面实现无缝的全隧道优化。
2026年一代GEM从孤立的人均域优化——如Facebook,Instagram,Reels——转向单一的中央基础模型。 它学习了Meta所有广告平台的数据和信号，在LLM尺度计算时弯曲了缩放曲线，并通过培训后技术提供强大的广告商结果，这些技术可以提供更大的耐久性预算，而不需要实时对所有广告候选人的评价——都放在一个更高效的培训堆上。 结果是在每一个表面和漏斗阶段同时获得广告性能的提高，更深刻地理解广告更符合人们所期待的。
GEM在Meta的广告表面和目标上提供了18.9%的转换改进率和12%的广告点击率(CTR)改进率，并在去年的Facebook、Instagram和场外转换中为目标用户提供了5.8%、4.1%和1.6%的转换提升率。 这些收益跨越上层隧道表面和深层隧道优化目标，促使广告商获得更大的收益。
GEM在以下四个领域推进推荐人的系统前沿：
1. 铺设放大曲线，使2.5x模型复杂
挑战：经验缩放定律表明，模型性能通过使用变压器模型架构的计算，参数和数据大小可以预测得到改进。 然而，由于巨大的不同特性和变化的分布，推荐器系统模型没有显示与变压器类似的缩放特性。
解决方案：我们建造了LLM风格的架构——昆仑，与最新的GPU硬件共同设计，使更大，更深的模型大幅提高成本效益，并使长文本学习超越用户参与历史。 我们观测到了LLM类的缩放定律在我们最大的培训前规模，2.5x模型复杂度增长从早期的GEM版本。 该图案反映了前沿LLMs：更多的能力和更长的背景在排名质量方面产生了新的收益，直接转化为业务成果。
2. 从GEM到用户阵容模型的有效情报转移
挑战：实时提供这种规模的模型费用太高，而且会延长时间，既会损害用户的经验，也会损害广告商ROI。
解决方案：结构化的表示，LoopFM，一种最先进后训练技术，将GEM的内部表示和顺序历史记忆转移到下游模型。 通过计算共享，其翻倍将智能传输与上一个版本部署的知识蒸馏一起，不增加计算成本或延迟。 我们还用推出弹性预计算(Solaris)来收获上游模型的收获。
3. 在Meta Ads Scale上加速使用大尺度跨扫描面，跨域学习的中央计算引擎
挑战：Meta的应用跨越了数十个表面——Facebook、Instagram、Reels、Treaties等等，每天有36亿人使用至少一个我们的应用。 每个表面和优化目标以前都有自己的独立预测系统。 随着平台在表面和目标上的增长，同时提高广告性能的挑战也随之增加。
解决方案：GEM通过中央基础模型加速一个单一，统一的计算引擎，该模型共同预测点击和转换跨越所有表面，包括Facebook和Instagram，以及跨越从场外转换到移动应用安装的每一个关键广告主目标。 在数据丰富的点击任务传输到数据共享转换任务上学到的泛泛表示。 一种新的免费数据混合和域专家框架的培训方法使这种多样化的大规模学习具有可预期的规模和更高的可靠性。
4. 将培训范围扩大4倍将E2E培训效率翻一番，覆盖数千个GPU
挑战：在这个尺度上，在数千个GPU中培训RecSys基础模型，需要精心安排计算、平行、交流和记忆。 今天的数据中心GPU及其软件堆栈大多被优化用于LLM工作量，而推荐工作量由于独特的数据特性而具有根本不同的外观，使得在规模上实现高GPU计算利用率极为困难。
解决方案：我们开发了低精度的自定义内核和与最新的GPU硬件和Meta的训练集群共同设计的新的5D并行战略，并优化了与基础设施相结合的模型架构，以便在数千GPU中高效扩展。 端到端培训效率翻了一番，达到20-25%的FLOPS(MFU)模式使用率，总培训量为4x FLOP，使得GEM可以在Meta的LLM级培训(参见我们关于LLM级培训GEM的同伴博客中的全部技术故事).
缩放法：与昆仑连接的边框缩放曲线
GEM 推出型新型LLM型示范建筑——昆仑。 与之前的版本(Interformer)相比，昆仑以相同的计算缩放方式交付2x性能增益。 昆仑的设计围绕以下创新：昆仑：
- 通过无缝地将终身序列模型与可伸缩的、针对具体建议的交叉交互连接，使架构框架标准化和统一。
- 通过与新一代硬件进行深度协同设计，采用可扩展架构，在几乎没有模型质量回归的情况下进行计算减少。
- 创新高效的序列规模，以离线序列基础设施处理较长的用户参与环境。
扩大建议范围的有效结构
GEM的昆仑架构通过赋予超长用户序列和大规模用户-项目交互功能，在前沿训练尺度上有效弯曲推荐器系统缩放曲线。 对于以下技术讨论，我们定义了两类特征：非序列特征(如用户人口统计，广告内容表述)，以及序列特征，带有一系列事件，每个特征由一组特征组成。
这个架构推出有三个基本进步：
1. 标准化和统一架构框架
我们推进了LLM风格的架构，以更好地捕捉推荐器系统模型的独特特征——具体来说就是各种特征和数据动态。
每个层包括：
- 注意：通过适应数据动态，定制注意力和向导网络，以捕捉用户行为中的长期依赖模式。
- 汇总部分：简化和加强汇总结构，从序列中提取关键用户的见解，有效连接序列和非序列结构。
- 交叉相互作用：扩展至专家混合(MoE)风格设计，使序列和非序列特征之间的复杂相互作用得以大规模建模。
2. 通过深层共同设计引入可扩展结构
我们与最新一代的硬件共同设计了这个架构，使之能更深入、更广阔地高效地发展：
–通过块状注意(Block Reception)：用块状的零散注意力取代完全的自留心，减少与顺序长度相关的复杂性，从四极到线性相对于窗口大小，只使用块状大小的倍数的窗口大小，对内核不带部分块大小。
- 通过建筑改造实现组件统一：每例Feed-Forward Networks(pFFN)是推出，以适应传统的FFN的每例分配。 通过通用点-生产注意(GDPA)，我们把PFFN部分重新调整为注意操作，通过将模型的许多注意类操作统一成一个单一的GPU内核来加速训练，最多提供2x前方加速。
- 通过专家平行主义进行可扩展的交叉互动：我们的交叉互动部分利用一个可扩展的、类似教育部的框架，采用模型-专家平行主义，使每个全球pu能够同时处理一组专家子网络。
3. 创新高效序列扩展
我们推出专用组件设计使超长序列可以计算：
组件跳过连接(CompSkip)：计算跳过机制有选择地激活每层的序列模块，并且只通过剩余连接输出路径。 通过有选择地去除特定组件，基于网络在重新激活后能够迅速恢复的假设，我们把FLOPS减少了43%，质量损失可忽略不计。
序列统一(OneSeq)：在事件特征汇总后，我们有选择地按时间顺序统一用户序列。 这使得一个单一的，统一的序列层可以处理长序列，即使单个事件具有不同的结构和特征。 在系统中，我们有不同的用户序列源，其中一些有不同的事件特征。 例如，购买序列中的事件可以通过ad ID和账户ID来描述，而邮递序列中的事件则由演员和对象ID来描述。 由于不同的事件特征，它们不能简单地从数据源级别统一。 它要求采用灵活的序列统一方法，以最大限度地提高GPU的效率，因为大尺寸的拉伸操作和尽量减少操作员的分散。
超长序列如何重要
新的建筑使超长序列变得可行。 GEM在生产中将长的接触环境长度缩放2x，并进一步使长的接触环境能够缩放到25x，显示了前沿规模的基础设施能力。 一路上，我们发现：
- 预测质量遵循有序列长度的动力定律。 这种关系是可以预测的，并且跨序列长度一致。
- 较大模型从较长的序列中获得的利益不成比例。 这创造了良性循环：缩放计算使模型更大，这使得更长的序列更有价值。 缩放序列长度使模型更加准确，这更有理由计算。
- 两项缩放法相辅相成，提供投资方向，使总体缩放法曲线不局限于简单的计算。
培训后：规模化的知识转让
基础模型的价值取决于其智能向实时服务于广告的数百个垂直模型的传输效果如何。 GEM通过结构代表与来自中央大脑的深层历史记忆，将推荐系统的后期培训革命化，实现知识蒸馏以外的转移有效性的2x. 此外，Async计算是推出到少数特定域的基础模型，将知识转移到垂直模型。
规模结构代表
知识衍射是有效的，但在带宽方面是有限的：下游模型只在每次培训过程中都得到GEM智能的压缩视图。 为了解决这个限制，我们引入了预计算的结构代表，如LoopFM，来理解和压缩中心大脑从基础模型到紧凑的代表，然后用深刻的历史记忆来构建。 它使下游模型能够从我们的大脑中学习到长长的背景。 它由全系统的下列主要组成部分组成：
- 理解和满足
本文由 AI宝盒自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Last year, we introduced GEM, the industry’s largest recommendation system (RecSys) foundation model. GEM established Meta’s first LLM-inspired paradigm for ads recommendation, driving major performance gains. Building on this foundation, our next goal is to further scale both compute and intelligence transfer with next-generation hardware, and expand this capability across Meta’s entire ads ecosystem. This spans dozens of surfaces and objectives, delivering seamless full-funnel optimization across awareness, consideration, and conversions.
The 2026 generation of GEM shifts from isolated per-domain optimizations — such as Facebook, Instagram, Reels — to a single central foundation model. It learns from data and signals across all of Meta’s ads platforms, bends the scaling curve at LLM-scale computation, and delivers strong advertiser outcomes through post-training techniques that provide a larger latency budget without requiring real-time evaluation of all ad candidates — all on a substantially more efficient training stack. The result is simultaneous ad performance gains across every surface and funnel stage, and deeper understanding that better matches ads to what people are looking for.
GEM has delivered 18.9% conversion improvement and 12% ad click-through rate (CTR) improvement across Meta’s ad surfaces and objectives, and additionally contributed 5.8%, 4.1%, and 1.6% conversion uplift on Facebook, Instagram, and off-site conversion for targeted users over the past year. These gains span both upper-funnel surfaces and deep-funnel optimization goals, driving stronger returns for advertisers.
GEM advances the frontier of recommender systems in the following four areas:
1. Bending the Scaling Curve, Enabling 2.5x Model Complexity
Challenge: Empirical scaling laws demonstrate that model performance predictably improves with compute, parameter, and data size using transformer model architecture. However, recommender system models do not exhibit similar scaling properties with transformer due to massive heterogeneous features and shifting distributions.
Solution: We built an LLM-style architecture — Kunlun, co-designed with the latest GPU hardware, to make larger, deeper models substantially more cost-efficient and to enable long-context learning over user engagement history. We observed an LLM-like scaling law at our largest pre-training scale with 2.5x model complexity growth from earlier versions of GEM. The pattern mirrors frontier LLMs: more capacity and longer context produce emergent gains in ranking quality that translate directly into business results.
2. Efficient Intelligence Transfer From GEM to User-Facing Models
Challenge: Serving a model at this scale in real time is prohibitively expensive and drives up latency, compromising both user experience and advertiser ROI.
Solution: The structured representation, LoopFM, a state-of-the-art post-training technique, transfers GEM’s internal representations and sequential historical memory to downstream models. Through computation sharing, it doubles the intelligence transfer together with the knowledge distillation deployed in the previous version, at no additional compute cost or latency. We also introduced elastic pre-computation (Solaris) to harvest the gains from the upstream model.
3. Accelerating the Central Compute Engine With Large-Scale Cross-Surface, Cross-Domain Learning at Meta Ads Scale
Challenge: Meta’s apps span dozens of surfaces — Facebook, Instagram, Reels, Stories, and more — with 3.6 billion people using at least one of our apps each day. Each surface and optimization objective previously had its own independent prediction system. As the platform grows in surfaces and objectives, so does the challenge of improving ad performance simultaneously.
Solution: GEM accelerates one single, unified compute engine through the central foundation model that jointly predicts clicks and conversions across all surfaces, including Facebook and Instagram, and across every key advertiser objective from off-site conversions to mobile app installs. The generalizable representations learned on data-rich click tasks transfer into data-sparse conversion tasks. A new training recipe with cost-free data mixing and the domain expert framework makes this heterogeneous, large-scale learning tractable, with predictable scale and enhanced reliability.
4. Scaling Total Training FLOPs by 4x While Doubling E2E Training Efficiency Across Thousands of GPUs
Challenge: Training a RecSys foundation model across thousands of GPUs at this scale requires careful orchestration of compute, parallelism, communication, and memory. Today’s data center GPUs and their software stacks are mostly optimized for LLM workloads, whereas recommendation workloads have a fundamentally different profile due to unique data characteristics that make it extremely difficult to achieve high GPU compute utilization while scale.
Solution: We developed custom kernels with low precision and new 5D parallelism strategies co-designed with the latest GPU hardware and Meta’s training cluster, and optimized model architecture with infrastructure together to scale efficiently across thousands of GPUs. End-to-end training efficiency is doubled to 20-25% model FLOPS utilization (MFU) with 4x total training FLOPs, enabling GEM to train at LLM-scale at Meta (see full technical story in our companion blog on Training GEM at LLM Scale).
Scaling Law: Bend Scaling Curve at Frontier Scale with Kunlun
GEM introduces a new LLM-style model architecture — Kunlun. Compared to the previous version (Interformer), Kunlun delivers 2x performance gains with the same computation scaling. Kunlun is designed around the following innovations:
- Standardize and unify architecture framework by seamlessly interleaving lifelong sequence modeling with scalable recommendation-specific cross-interaction.
- Introduce scalable architecture through deep co-design with the latest generation of hardware, and computation reduction with almost no model quality regression.
- Innovate efficient sequence scaling to process longer user engagement contexts with offline sequence infrastructure.
Efficient Architecture for Scaling Recommendations
GEM’s Kunlun architecture effectively bends the recommender system scaling curve at frontier training scales by empowering ultra-long user sequences and massive user-item interaction features. For the following technical discussion, we define two types of features: non-sequence features (such as user demographics, ad content representations), and sequence features with a sequence of events, each composed of a set of features.
This architecture introduces three fundamental advancements:
1. A Standardized and Unified Architecture Framework
We advanced the LLM-style architecture to better capture the unique characteristics of recommender system models — specifically, heterogeneous features and data dynamics.
Each layer consists of:
- Attention: customize the attention and feed-forward networks to capture the long dependency pattern in user behavior by adapting to data dynamics.
- Summarization Component: simplify and enhance the summarization architecture to extract critical user insights from sequences, effectively bridging sequence and non-sequence structure.
- Cross Interaction: expand to the mixture-of-experts (MoE) style design, enabling large-scale modeling of complex interactions between sequence and non-sequence features.
2. Introducing Scalable Architecture Through Deep Co-design
We co-designed the architecture with the latest generation of hardware to allow it to grow deeper and wider efficiently:
- Computation Complexity Reduction via Block Attention: Replacing full self-attention with block-aligned sparse attention reduces the complexity associated with sequence length from quadratic to linear relative to window size, and uses only the window size that is a multiple of block size, friendly for the kernel without partial block size.
- Component Unification via Architecture Reformulation: Per-Example Feed-Forward Networks (pFFN) was introduced to adapt the traditional FFN to per-example distribution. We reformulated the pFFN component into attention operations through Generalized Dot-Product Attention (GDPA), speeding up training by targeting at unifying the model’s many attention-like operations into a single GPU kernel, delivering up to 2x forward speedup.
- Scalable Cross interaction through Expert Parallelism: Our cross-interaction component utilizes a scalable, MoE-like framework that employs model-expert parallelism, allowing each GPU to process a subset of expert sub-networks in parallel.
3. Innovating Efficient Sequence Scaling
We introduced specialized component designs to make ultra-long sequences computationally tractable:
Component Skip Connection (CompSkip): The computational skip mechanism selectively activates sequence modules in each layer, and only routes outputs through residual connections. By selectively deactivating the specific components, which is based on the assumption that the network can recover quickly after re-activating, we reduce FLOPS by 43% with negligible quality loss.
Sequence Unification (OneSeq): We selectively unify user sequences chronologically after the event feature aggregation. This allows a single, unified sequence layer to process long sequences, even when individual events possess heterogeneous structures and features. In the system, we have different user sequence sources, some of which have different events features. For example, an event in a purchase sequence can be described by ad ID and account ID, while the event in a post sequence is described by actors and object ID. They cannot simply be unified from the data source level due to the different set of event features. It requires the flexible sequence unification approach to maximize GPU efficiency because of large-size tensor operations and minimizing operator fragmentation.
How Ultra-Long Sequences Matter
The new architecture makes ultra-long sequences feasible. GEM scaled the long engagement context length by 2x in production, and further enabled the capability of scaling long context to 25x, demonstrating infrastructure capability at frontier scale. Along the way, we discovered:
- Prediction quality follows a power law with sequence length. The relationship is predictable and consistent across sequence lengths.
- Larger models benefit disproportionately from longer sequences. This creates a virtuous cycle: scaling compute makes the model bigger, which makes longer sequences more valuable. Scaling sequence length makes the model more accurate, which justifies more compute.
- The two scaling laws reinforce each other, offering investment directions to bend the overall scaling law curve beyond simply adding compute.
Post-Training: Knowledge Transfer at Scale
The value of the foundation model depends on how effectively its intelligence transfers to the hundreds of vertical models that serve ads in real time. GEM revolutionizes post training for recommendation systems through the structure representation with the deep historical memory from the central brain, achieving 2x of transfer effectiveness beyond knowledge distillation. Furthermore, async computation is introduced to a few domain-specific foundation models to transfer the knowledge to vertical models.
Structure Representation at Scale
Knowledge distillation is effective but limited in bandwidth: downstream models only receive a compressed view of GEM’s intelligence at each training process. To address the limitation, we introduce the precomputed structure representation, such as LoopFM, to understand and compress the central brain from the foundation model into a compact representation, and then structure it with deep historical memory. It enables the downstream models to learn the long context from our central brain. It consists of the following major components across the system:
- Understand and Comp
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.