自从理查德·萨顿于2019年发表有影响力的散文后，苦痛课的概念指导了AI中的许多现代进步。 这一概念支持利用计算和以规模学习的一般方法，尽可能不出现人为设计的前科。 我们的解释是，从本质上讲，智能是学习原始数据：自主地寻找规律，构建抽象，从混乱的低层次投入中发展出意义，理想的是端到端，尽可能地提高数据效率。 随着AI发展到科学，机器人和生物学等复杂领域 — — 数据稀缺，而且人类往往难以理解，更不用说工程师了 — — 将这一哲学内部化，以开发出更通俗的学习算法比以往任何时候都重要。
语言模型或许是AI今天最大的成功故事，它们的能力主要归功于一些简单的算法的缩放，如变形器架构，下接的预测，强化学习，以及测试时间的计算。 然而，它们仍然依赖于手动的编码步骤，如活化剂等，这严重地阻碍了它们从基础数据中学习端到端的能力。 我们认为，开发字节级语言模式是AI进步的基础 — — 不仅是改善独立的LLMs，也是一般学习方法的试金石。 如果我们甚至不能从字符和字节中很好地学习语言，那么我们在下一个领域有什么权利取得进展？
2025年，我们发展出分级网络(H-Nets)的概念，它不仅解决了语言模型中的象征性化问题，而且朝着更能从不同模式的原始数据中学习的更通用的结构迈出了一步。 自此我们大大改进了H-Net的技术，并讨论影响。
我们将会探索H-Net的缩放定律及其定性行为。
- 我们为象征性变形器制作一个标准缩放阶梯，并显示仔细控制的1级H-Net纯粹通过有学问的符号化提供有意义的计算效率增益(可达3-5x).
- 我们显示在扩大更灵活的二相H-Net方面取得的经验结果，它比开源模型的效绩要好，其效率得到极端提高(10x减计算和数据)。
- 我们分析H-Net所学的规律，它们显示出语言模型中以前所未见的丰富多彩的新行为，如内文符号化。
- 我们讨论H-Net如何超越语言建模，在生物学和愿景方面不断取得研究成果，显示作为跨模式的一般结构的希望。
我们将会更深入地探索更多模式
总体而言，我们的成果提供了证据，证明H-Net是一个基础架构，真正符合痛苦的教训，直接从原始数据中学习意义，并大大提高效率和可扩展性。 这个研究议程是推动AI如何从所有数据中学习的重要机会
等级网络(H-Nets)
我们推出分级网络(paper, blog)去年作为一个新的端到端架构，简言之：
术语的灵感来源于认知心理学中现存的分解概念，这是人类智能的重要组成部分。 我们认为，对人工智能的前沿也有必要。
在语言建模的背景下，这个想法是构建出新的学习原始，使模型可以将低等数据递归地分解为更高层次的意义：字符为词，词为词，词为概念。
就这一职位而言，人们大多可以认为，等级网络取代了象征性。 H-Net 并不是从字节到符号的预处理数据(固定的字节块)，而是在模型内直接动态地执行块。 此外，H-Nets的签字特征是，它们按等级进行这种分门别类的端到端。 相形之下，标准标志性语言模型甚至其它替代方案(有些将在《邮政报》二版中看到)使用多个培训阶段
细节：切入和BPE
语言模型不会直接读取文字，一个自然词汇是256个字节值，每个UTF-8字节产生一个符号。 然而，字节级序列对于标准语言模型的处理是漫长而昂贵的。 通过将字节序列压缩成经常出现的块来缓解这个问题。 最常见的方法是"字节平面编码(BPE)"从256字节开始，并反复地将固定训练符上最常出现的相邻的一对相接符合并，直到词汇达到设定大小(例如Llama-3). 在我们的数据组合上，每个BPE令牌覆盖了大约4.57字节，使序列比字节级要短得多。 然而，这种压缩是固定的，而象征性化仍然是用手设计而不是学习的语言模型管道的最后一部分。
我们将会看到，即使在取代象征化的狭义背景下，除了示意外，H-Net作为一个基础架构还有广泛的用途：
- H-Net可以被递归地嵌入，从而可以建立多层次的抽象。 这个员额的主要结果是比起开源基线，一个更大的两级分级网络。
- H-Net可以运行在任何类型的数据上，而不仅仅是字节，而不仅仅是离散的数据。 与此相反，对连续数据进行标记是一项严重的挑战。 在文章的最后部分，我们将预览一些未来运用H-Net的研究成果
图1说明了一个2相H-Net，并定义了模型的主要组成部分：编码器，主网络和解码器。 我们有时把编码器/解码器统称为外网和主模型为内网。
我们计划推出H-Net，我们将会更深入地评论这篇文章中与早期架构的不同之处。
第1部分：生物防护设备及H-Net扩能梯
与许多建筑论文一样，关于分级网络的原始论文侧重于调整特定固定尺寸和数据地平线的语言模型(例如从FineWeb-Edu得到的100B令牌上的~1.3B-参数模型). 虽然它们看起来很有希望，但缺少一个小作品是缺乏计算-优化的缩放法，现在是衡量预先训练创新的标准方法。 例如，Kimi在Kimi Delta Content(KDA)上的流行创新和注意力后遗症分别测量出1.16x和1.25x的计算乘数。
为了验证我们改进后H-Net架构的成果，我们首先着手建造一个现代化的LLM缩放梯子，然后用一个精心控制的分级网络来取而代之。
- 我们建立了一个培训混合体，用于在平等条件下比较字节平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面
——我们根据标准做法为基线的BPE变形器语言模型搭起一个缩放梯子，复制了文献的成果。
- 然后我们把每道跑道转换成一个字节级的H-Net，作为倒置替换，再使用BPE主干架构和超参数。
这种配方直接显示验证损失(1.3x)的收效更强，代码(2x)等子域的增益增加，并改进出发行(评价为2x，域外代码可达5x).
数据控制的比较
由于字节平面模型和BPE模型在不同颗粒上处理相同的数据，所以需要额外小心来控制方法。
数据混合：我们的培训数据集由70%的英语，20%的代码，5%的数学和5%的多语种组成，用字节数衡量。
数据处理：每个样本都作为字节和BPE符进行处理，在相同的边界上进行分拆；因此，字节和BPE处理的数据集都包含与对齐的文档边界完全相同的文本。
度量：我们使用标准比特-每字节(bpb)度量度来测量相邻的预测损失，在不同的表示器之间实现常态化。
公司内部的 BPE 代号器：LLMs一般使用现成的 BPE 代号器；H-Nets的一个优点是在训练分发上直接学习分块。 为了格外小心，我们还训练了BPE仪表器我们用它来进行BPE的缩放定律，最终在验证损失和下游电压上产生大约1.1x计算增益(因此H-Net相对于标准的现成活化器
BPE 的活字标注器尺寸为 128k，内置活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活化活以公司内部的BPE标注器表示，培训数据集由64.7%的英语、23.7%的代码、6.1%的数学和5.5%的多语种组成。
B. 建立一个强有力的BPE基线
对于我们的基线来说，我们适合一个经过良好调整的缩放定律一个标准的BPE变形器。 为此，我们仿照Kimi和MAI等现代的缩放配方，构建出一个缩放阶梯，用L层数来充分定义了一组模型。 我们把隐藏的维度D设定为128倍，这样一来，视比D/L约为75. 对于所有模型，我们使用多头注意，头数等于地层数，固定头尺寸为128. 对于MLP，我们使用中间维度为3.6D的SwiGLU地层，再四舍五入到128.1的相近倍数。
然后，我们把计算-最佳的缩放定律与标准异同FLOP扫描方法相匹配。
所有运行都使用有固定折叠热和衰变的Muon优化器和WSD学习速率表。
- 我们调整了批量大小和学习率，并安装了超参数缩放定律，以推导到较大的定律。
我们仔细计算了所有模型中的FLOP，包括所有序列混合层和LM头。 对于软马克斯的注意，我们也只计算出因果FLOP，因为由于四相复杂，过量计算出非因果FLOP会惩罚处理较长序列的字节级模型。
等效的 H- Net 梯级
因为H-Net 超过字节的字节与标准标注模型大相径庭，移动的部位如此之多：分出分出多少个阶段，相应的压缩速率，每个阶段的架构配置(outer vs 内网络)，以及每个部分的超参数。
我们决定研究一个更狭义的设定：H-Nets如何与BPE模型进行比较，如果仅限于被解释为纯粹取代了语言模型的象征性部分的话？
为此，我们把BPE梯级的每道跑道转换为字节一级H-Net. 一个1级的H-Net可以被看作一个具有已学习到的符号化的标准语言模型：内网络运行于块上(它扮演着符号的作用)，外网络则发挥动态符号化的作用，将字节编码为块并解码回。 每个径都非常小心地匹配：
- H-Net的内部网络
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Since Richard Sutton’s influential essay in 2019, the concept of the bitter lesson has guided much of modern progress in AI. This concept espouses general methods that leverage computation and learn with scale, as free of human-engineered priors as possible. Our interpretation is that, fundamentally, intelligence is about learning from raw data: autonomously finding patterns, building abstractions, and developing meaning from messy low-level inputs, ideally end-to-end and as data-efficiently as possible. As AI advances into complex domains such as science, robotics, and biology – where data is scarce and often difficult for humans to understand, let alone engineer for – internalizing this philosophy to develop more general learning algorithms is more important than ever.
Language models are perhaps AI’s biggest success story today, and largely embrace the bitter lesson. Their abilities are largely attributable to scaling a few simple algorithms such as the Transformer architecture, next-token prediction, reinforcement learning, and test-time computation. However, they still rely on hand-crafted encoding steps such as tokenizers, which severely hinder their ability to learn end-to-end from the underlying data. Towards unlocking another step change of scaling, we believe that developing byte-level language models is foundational for AI progress – not just for improving standalone LLMs, but also as a litmus test for general learning methods. If we can’t even learn language well from characters and bytes, what right do we have to make progress on the next frontier?
In 2025, we developed the concept of hierarchical networks (H-Nets), which not only addressed tokenization in language models, but took a step toward even more general-purpose architectures that could learn from raw data across different modalities. We’ve since significantly improved the technology of H-Nets and are excited to share their capabilities! In this series of two posts, we’ll dig into new results and discuss the implications.
In this first post of a two-part series, we’ll explore scaling laws for H-Nets and their qualitative behaviors.
- We produce a standard scaling ladder for tokenized Transformers, and show that carefully controlled 1-stage H-Nets provide meaningful compute efficiency gains (up to 3-5x) purely through learned tokenization.
- We show empirical results from scaling up a more flexible 2-stage H-Net, where it outperforms a matched open-source model with extreme efficiency improvements (10x less compute and data).
- We analyze the patterns that H-Nets learn, where they show a rich diversity of new behaviors previously unseen in language models, such as in-context tokenization.
- We discuss how H-Nets go beyond language modeling, with ongoing research results in biology and vision, showing promise as a general architecture across modalities.
In the next post, we’ll do a deeper exploration of a larger range of models to understand better why H-Nets can be so much more efficient for language modeling.
Altogether, our results provide evidence that H-Nets are a foundational architecture that truly aligns with the bitter lesson, learning meaning directly from raw data and significantly improving efficiency and scalability. This research agenda represents a major opportunity to advance how AI can learn from all data, and we couldn’t be more excited to push the frontier!
Hierarchical Networks (H-Nets)
We introduced hierarchical networks (paper, blog) last year as a new end-to-end architecture that models multiple levels of abstraction in data, showing promise in language modeling and beyond. In a nutshell:
The terminology was inspired by the existing concept of chunking in cognitive psychology, which is an essential part of human intelligence. We believe it is also necessary for the frontier of artificial intelligence.
In the context of language modeling, the idea is to build new learning primitives that allow models to recursively chunk low-level data into higher-level meaning: characters into words, words into phrases, phrases into concepts.
In the context of this post, for the most part, one can think of hierarchical networks as replacing tokenization. Instead of pre-processing data from bytes into tokens (fixed chunks of bytes), H-Nets perform the chunking dynamically directly inside the model. Furthermore, the signature characterization of H-Nets is that they hierarchically perform this chunking end-to-end. In contrast, standard tokenized language models and even other alternatives (some of which we’ll see in Post II) employ multiple stages of training, and the process of chunking / tokenizing the data is performed entirely offline.
Details: Tokenization and BPE
Language models don’t read text directly but instead take as input a sequence of tokens from a fixed vocabulary. A natural vocabulary is the 256 byte values, producing one token per UTF-8 byte. Byte-level sequences, however, are long and expensive for standard language models to process. Tokenization aims to alleviate this problem by compressing the byte sequence into frequently occurring chunks. The most common approach, Byte Pair Encoding (BPE), starts from the 256 bytes and repeatedly merges the most frequent pair of adjacent tokens on a fixed training corpus, until the vocabulary reaches a set size (e.g. 128k for Llama-3). On our data mix, each BPE token covers about 4.57 bytes, making sequences much shorter than at the byte level. However, this compression is fixed, and tokenization remains one of the last parts of the language-modeling pipeline that is designed by hand rather than learned.
We’ll see in this post series that even in the narrower context of replacing tokenization, the end-to-end property of H-Nets provides real performance gains. Beyond tokenization, H-Nets have broad utility as a foundational architecture:
- H-Nets can be nested recursively, allowing for multiple levels of abstraction to be built. The main result of this post is a larger two-stage hierarchical network matched against an open-source baseline.
- H-Nets can operate over any type of data, not just bytes, and not just discrete data. In contrast, tokenizing continuous data is a serious challenge. In the final part of this post, we’ll preview some future research applying H-Nets beyond text.
Figure 1 illustrates a 2-stage H-Net and defines the main components of the model: the encoder, main network, and decoder. We sometimes call the encoder/decoder together the outer networks and the main model the inner network.
Throughout this post, we show results with a much improved version of H-Nets, which we plan to release. We’ll comment more on differences from the earlier architecture later in this post.
Part 1: BPE and H-Net Scaling Ladders
As with many architecture papers, the original paper on hierarchical networks focused on tuning language models of particular fixed sizes and data horizons (e.g. ~1.3B-parameter models on 100B tokens from FineWeb-Edu have been very common). Although they looked very promising, a missing piece was the lack of compute-optimal scaling laws, now a standard way to measure pretraining innovations. For example, Kimi’s popular innovations on Kimi Delta Attention (KDA) and attention residuals measured compute multipliers of 1.16x and 1.25x, respectively.
In order to validate the gains of our improved H-Net architecture, we first set out to build a modern LLM scaling ladder, and then replaced its tokenization components with a carefully controlled hierarchical network.
- We set up a training mixture for comparing byte-level and BPE scaling laws on equal terms, carefully matching the processing and even re-training a BPE tokenizer for the baseline.
- We set up a scaling ladder for a baseline BPE Transformer language model according to standard practices, reproducing results from the literature.
- We then converted each rung to a byte-level H-Net as a drop-in replacement, reusing the BPE backbone architecture and hyperparameters.
This recipe directly showed stronger results on validation loss (1.3x), with gains increasing on subdomains such as code (2x), as well as improving out of distribution (2x on evaluation, and up to 5x on out-of-domain code).
A data-controlled comparison
Since byte-level and BPE models process the same data at different granularities, extra care is needed to keep the methodology controlled.
Data mixture: Our training dataset consisted of 70% English, 20% code, 5% math, and 5% multilingual, measured by the number of bytes.
Data processing: Every sample is processed as both bytes and BPE tokens, splitting at the same boundaries; both byte- and BPE- processed datasets therefore contain exactly the same text with aligned document boundaries.
Metric: We use the standard bits-per-byte (bpb) metric for measuring next-token prediction loss in a way normalized between different tokenizers.
In-corpus BPE tokenizer: LLMs generally use off-the-shelf BPE tokenizers; an advantage of H-Nets is that they learn their chunking directly on the training distribution. To be extra careful, we also trained a BPE tokenizer on the corpus. We used this for the BPE scaling laws below, which ended up resulting in about a 1.1x compute gain on both validation loss and downstream evals (so the H-Net would have an even greater compute multiplier against a standard off-the-shelf tokenizer!).
The BPE tokenizer’s vocab size is 128k, matching the Llama-3 tokenizer that we were working off of. The in-corpus tokenizer compressed the dataset slightly more, increasing the average BPE token length from 4.45 (Llama-3 tokenizer) to 4.57. Measured in tokens of the in-corpus BPE tokenizer, the training dataset comprised 64.7% English, 23.7% code, 6.1% math, and 5.5% multilingual.
Creating a robust BPE baseline
For our baseline, we fit a well-tuned scaling law for a standard BPE Transformer. To do this, we constructed a scaling ladder, following modern scaling recipes such as Kimi and MAI, fully defining a family of models by the number of layers L. We set the hidden dimension D to a multiple of 128 such that the aspect ratio D/L is approximately 75. For all models, we use multi-head attention with the number of heads equal to the number of layers and a fixed head dimension of 128. For the MLPs, we use SwiGLU layers with the intermediate dimension set to 3.6D, again rounding to the nearest multiple of 128.1
We then fit compute-optimal scaling laws with the standard iso-FLOP sweep methodology.
- All runs use the Muon optimizer and WSD learning rate schedule with fixed-fraction warmup and decay.
- We tuned batch size and learning rate on selected rungs and fitted hyperparameter scaling laws to extrapolate to larger ones.
- We counted FLOPs carefully in all models, including all sequence-mixing layers and the LM head. For softmax attention, we also count only causal FLOPs, as, due to the quadratic complexity, over-counting non-causal FLOPs penalizes byte-level models which process longer sequences.
An equivalent H-Net ladder
Because H-Nets over bytes are so different from standard tokenized models, it’s highly nontrivial to compare them directly. There are so many moving parts: how many stages of chunking, the respective compression rates, the architecture configurations of each stage (outer vs. inner networks), and the hyperparameters for each component.
We decided to investigate a narrower setting: how do H-Nets compare against BPE models, if restricted to being interpreted as purely replacing the tokenization portion of the language model?
To do this, we converted each rung of the BPE ladder to a byte-level 1-stage H-Net. A 1-stage H-Net can be viewed as a standard language model with learned tokenization: the inner network operates over chunks (which play the role of tokens) and the outer networks perform the role of dynamic tokenization, encoding bytes into chunks and decoding them back. Each rung is matched very carefully:
- The H-Net’s inner network
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.