包括最近一篇关于当代LLMs控制推理水平机制的文章。 理性模型仍然一如既往地具有相关性。 事实上，今天几乎所有的当代LLMs都是推理模型或提供推理模式。
对于那些对推理模型如何工作感兴趣的人，包括推论缩放，强化学习，和分解，我写了最近我所写的"构建理性模型"(出自"分解")，这个模型从PyTorch的地上执行这些，根据我的经验，这仍然是理解和学习事物如何真正工作的最好方法。
在过去几个月里，我还坐下来录制了我想与你们分享的讲座。 因此，下面，你会找到8个推理模型-从scratch编码视频(内容11小时).
我们从一个经过预先训练的LLM开始，并在Python和PyTorch执行评价和改进的方法。 因此，如果你喜欢我的编码LLMs 从去年的"地上"课程，这基本上是续集(但你也可以不以先前的课程为先决条件观看).
下个月我将会回来，在此期间，我希望你能享受编码！
LLMs，推理模型，和agents的关系和相接方式
我经常被问及推理模式与今日的AI agents有何关系
一个推理模型本质上是从传统的LLM升级而来，即通过可核查的奖励(和可选的推论缩放技术)的强化学习来增加更多的后训练，使其更有能力解决更具有挑战性的问题。 经典地，推理模型是为数学和代码所开发的，但能力也可以被概括到其他问题领域。
一个推理模型然后作为agent拉带内部的引擎发挥作用，其中agent拉带是一个围绕LLM/推理模型的软件或工具框架，可以解决更具有挑战性的问题(bigger编码任务，旅行预订，以及所有其他类型的个人助理任务). 牵引装置的任务是复杂的上下文管理并让LLM获得工具。
图1：常规LLMs，推理模型和agent牵引装置的引擎类比。 图片来自我编码Agent文章的组件。
在上图所描绘的这种类比中，模型是引擎，而agent将新增用上让我们用于各种事物的机械(作为从简单的聊天器界面上升级). agent既可以使用常规的LLM，也可以使用推理模型，但推理模型是比较典型的选择，因为它们的能力更强。 (如上所提到，大多数流源LLMs是具有可变推理努力设置的推理模型。 )
因此，底线是推理模型作为"引擎"，仍然是理解现代LLM agents的重要组成部分。
上述专题的配套资源如下：
- 构建大语言模型(取自斯克勒克)书籍和视频
- 构建一个理性模型(取自"刮")书(和下面的视频)
- 关于Substack的编码Agent文章的组成部分
以下的视频专注于开发能够为这种系统提供动力的推理模型。
范围和先决条件
推理-从scratch视频聚焦于一个小型的预训Qwen3模型，我们自己在Python和PyTorch中执行推理方法。 从一个经过预先训练的模型开始，让我们集中关注有趣的部分：评价，推断时间的缩放，强化学习，和分馏。
关于先决条件，我建议对Python和PyTorch感到舒适。 我将推理材料设计为独立的内容，因此之前的"Build A Ligh Language Model"(从Scratch)内容被推荐，但并没有严格要求。
计算资源时，可以在CPU上通过早期的文本生成和评价实例来工作。 对于后期的实验，特别是训练，一个GPU使工作更加实用。 首个视频涵盖了设置和一些更多的建议；存储器还包括硬件和环境建议。
图2：涵盖的专题概述。
伴书
这些视频与我的书"构建一个理性模型"(From Scratch)相伴。 如果你想更详细地和以你自己的速度阅读解释，通过练习来工作，等等，我在此推荐这本书作为伴奏。
对于最深入的学习经验，我的建议是读出一章，通过它的代码和相应的视频一起工作，然后尝试练习。
本作取自"曼宁"和"Amazon"，印刷版取自"彩相"!.
图3 Build a Reasoning Model(从斯克勒克)印刷本的随机摘录。
感谢所有收到副本或推荐给朋友的人。 你的支持帮助我不断制作这些材料。
视频
最后，更简短、更注重概念的概述，我还建议我关于链接的AI理性模型课程在学习中(在通过该链接访问时24小时应该免费)
课程约90分钟，并解释了推理模型是如何工作的，涵盖了推论-时间缩放，强化学习，和分馏的主要思想。 如果你喜欢主要想法
为了了解这篇文章的主要内容，这里是八大讲座视频。
1. 动机和代码设置
第一部影片以解释常规LLMs，推理模型与agents的关系为开头。 接下来，视频的主要话题是建立编码环境。 这包括安装Python和PyTorch的依赖(与uv)等。 我还讨论了硬件选择和共同设置问题。
涵盖的专题
- 常规LLMs、推理模型和agent牵引装置之间的关系
- 通过执行守则学习的益处
- 这一系列视频的范围
- 安装 Python 依赖
- 选择硬件并检查CUDA或Apple硅支持
- 使用具有VS代码的远程GPU，并排除环境出问题
观看视频 1 —— 设置指令
2. 装入基本模型、文本生成和 KV 缓存
在即将到来的视频中加入推理方法之前，我们需要一个工作基础模型。 与其从零开始训练它，因为这将是另一本完整的书(参见Build A Lige Language Model(从"斯克勒克"开始))，不如从预先训练过的模型开始。
在这里，我们加载Qwen3 0.6B型机的前训练重量。 然后，我们通过流出式文本生成器实施文本生成过程，并添加了KV缓存和模型编译来提高文本生成效率。
这些执行情况为我们提供了基线和基本内容，我们将在以下视频中评价和改进。
涵盖的专题
- 将文本编码为符号并重新编码为文本
- 下载Qwen3重量并装入选定计算设备上的模型
- 翻过收音机形状、记录和下一个预测
- 实施贪婪的解码和流出文本生成
- 处理序列终结符并测量生成速度
- 添加 KV 缓存以再利用注意键和值
- 将 KV 缓存与火炬相混合。 编译并比较硬件性能
观看视频 2 —— 第2章代码
3. 建立评价和强化学习核查机构
在这个讲座中，我们一步一步地建立一个数学答案验证器，从答案提取到正常化和检查数学等同性。 然后用它来评估MATH-500上的模型。
我们用这个验证器来评价基模在MATH-500评价数据集上的性能。
然而，核查员在系列的后期也有第二个工作：为强化学习提供可核查的奖励信号(RLVR).
涵盖的专题
- 核查人介绍及其如何融入LLM评价方法(其他方法载于本书附录)
- 将回答核查作为RLVR的奖励信号(详见第6章)。
- 核查者流动，包括提取最后答案、使答案正常化并检查数学等同性
- 为MATH-500建造一个回答分级器和评价循环
- 比较MATH-500的基础和推理模型
观看视频 3 章码
4. 推断时间尺度：采样和自我一致性
现在，我们探索改进答案的方法，同时保持模型重量的固定。 但在执行具体的推论分级技术之前，我们首先执行方法，从同一个快取生成不同的答案(这是专有LLM聊天器的标准做法，如ChatGPT).
具体地说，我们实施温度缩放、多分子取样和上p过滤，以产生各种不同的反应。 然后，我们通过自我一致性，利用多数人对其最后答复的投票，将多种反应结合起来。
我们还审查了答案准确性和产生更多候选人所需的额外计算之间的取舍。
涵盖的专题
- 比较在培训和推论期间如何额外计算效益模型的业绩
- 温柔的引言推论通过思维链的提示
- 实施温度缩放和多名抽样，以产生不同的答案
- 添加上p过滤器以控制不同答案的质量
- 通过自我一致来综合候选人的答案，这是最受欢迎的推论缩放技术之一
- 评价MATH-500的准确性与额外推论计算的费用
观看视频 4 — 第4章代码
5. 推断时间尺度：评分和自我改进
在之前的视频的基础上，我们执行评分功能，以比较来自同一提示的多个答案。 我们还可以使用这些评分方法，在自我一致性上断绝关系(在前一次讲座中涵盖).
这里覆盖的评分方法包括计算符和序列日志概率(这在后期的强化学习章节中也将相关). 然后我们实施一个自我完善的循环，其中模型批评一个答案并试图修改。
评价让我们检查这些额外步骤对我们的模式有多大用处。
涵盖的专题
- 利用评分功能比较多个答案并断开联系
- 实施简单的高压计分器
- 计算符号概率和序列可能性(也与以下讲座有关)
- 利用日志概率来提高数值稳定性
- 为提出批评和订正答案建立自我完善循环
- 评估MATH-500上的自我改进
观看视频 5 章代码 5
6. 强化学习：为RLVR实施GRPO
在这次讲座中，我们着手实施一种培训技术，称为“强化学习与可验证奖励”。 我们首先实施简化版的"集团相对政策优化"(GRPO)，生成各组回复，与我们的核查员评分，并在培训期间使用他们的相对奖励来更新模型重量。
涵盖的专题
- 生成推理痕迹并理解答案长度和解决问题能力之间的关系
- 讨论和比较两个重要的LLM培训后方法：强化学习与人类反馈(RLHF)和强化学习与可验证奖励(RLVR)
- 将核查分数转化为奖励和相对优势
- 计算序列日志概率和简化的GRPO损失
- 将训练循环与伐木和检查站储蓄结合起来
- 评价检查站并检查培训稳定性和记忆需要
观看视频 6 章代码
7. 强化学习：跟踪、剪接
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
I have written a lot about reasoning models here over the last two years on Ahead of AI, including a recent article on mechanisms to control reasoning effort levels in contemporary LLMs. Reasoning models are still as relevant as ever. In fact, almost all current-generation LLMs today are reasoning models or offer a reasoning mode.
For those who are interested in how reasoning models work, including inference scaling, reinforcement learning, and distillation, I wrote my recent Build Reasoning Model (From Scratch), which implements these from the ground up in PyTorch, which, in my experience, is still the best way to understand and learn how things really work.
Now, in the last few months, I also sat down and recorded accompanying lectures I wanted to share with you. So, below, you will find eight reasoning model-from-scratch coding videos (11 hours of content).
We start with a pre-trained LLM and implement methods to evaluate and improve it in Python and PyTorch. So, if you enjoyed my Coding LLMs from the Ground Up course last year, this is essentially the sequel (but you can also watch it without the prior course as a prerequisite).
I’m currently taking a short break and will be back soon with more exciting LLM research roundups next month. In the meantime, I hope you enjoy coding along!
How LLMs, reasoning models, and agents relate and fit together
As a small side note, and since I often get asked how reasoning models relate to today’s AI agents, here’s a short section on this before getting started with the videos.
A reasoning model is essentially an upgrade from a conventional LLM, that is, adding more post-training via reinforcement learning with verifiable rewards (and optionally inference scaling techniques) to make it more capable of solving more challenging problems. Classically, reasoning models are developed for math and code, but the capabilities can generalize to other problem domains as well.
A reasoning model then functions as the engine inside an agent harness, where the agent harness is a software or tooling framework around the LLM/reasoning model to tackle even more challenging problems (bigger coding tasks, travel booking, and all other sorts of personal assistant tasks). The task of the harness is sophisticated context management and giving the LLM access to tools.
Figure 1: An engine analogy for conventional LLMs, reasoning models, and agent harnesses. Figure from my Components of a Coding Agent article.
In this analogy depicted in the figure above, the model is the engine, and the agent harness adds the machinery that lets us use it for various things (as an upgrade from a simple chatbot interface). An agent can use either a conventional LLM or a reasoning model, but reasoning models are the more typical choice as they are more capable. (As mentioned above, most current-gen LLMs are reasoning models with variable reasoning effort settings.)
So, the bottom line is that reasoning models, as the “engine,” remain an important part of understanding modern LLM agents.
The resources accompanying these topics shown above are:
- The Build a Large Language Model (From Scratch) book and videos
- The Build A Reasoning Model (From Scratch) book (and videos below)
- The Components of a Coding Agent article on Substack
The videos below focus on developing the reasoning model that can power such a system.
Scope and prerequisites
The reasoning-from-scratch videos focus on a small pre-trained Qwen3 model, where we implement the reasoning methods ourselves in Python and PyTorch. Starting with a pre-trained model lets us focus on the interesting part: evaluation, inference-time scaling, reinforcement learning, and distillation.
Regarding the prerequisites, I recommend being comfortable with Python and PyTorch. I designed the reasoning materials as standalone content, so the previous Build A Large Language Model (From Scratch) content is recommended but not strictly required.
Compute-resource-wise, you can work through the early text-generation and evaluation examples on a CPU. For the later experiments, especially training, a GPU makes the work much more practical. The first video covers the setup and some more recommendations; the repository also includes hardware and environment recommendations.
Figure 2: An overview of the topics being covered.
The companion book
These videos accompany my book, Build a Reasoning Model (From Scratch). If you like to read through the explanation in more detail and at your own pace, work through exercises, and so on, I recommend the book as a companion here.
For the most in-depth learning experience, my suggestion is to read a chapter, work through its code alongside the corresponding video, and then try the exercises.
The book is available from Manning and Amazon, and the print edition is in color!
Figure 3: A random excerpt from the Build a Reasoning Model (From Scratch) print book.
Thank you to everyone who has picked up a copy or recommended it to a friend. Your support helps me keep creating these materials.
The videos
As a last, quick side note before getting to the main videos below: For a shorter, more conceptual overview, I also recommend my AI Reasoning Models course on LinkedIn Learning (it should be free for 24h when accessing it via the link there, which shouldn’t then require a LinkedIn Premium subscription).
The course is about 90 minutes and explains how reasoning models work, covering the main ideas behind inference-time scaling, reinforcement learning, and distillation. It’s a good optional preview if you’d like the main ideas without working through the full implementations below.
To get to the main part of this article, here are the eight main lecture videos.
1. Motivation and code setup
The first video starts with an explanation of the relationship between conventional LLMs, reasoning models, and agents. Next, the main topic of the video is setting up the coding environment. This includes installing the Python and PyTorch dependencies (with uv), among others. I also discuss hardware choices and common setup issues.
Topics covered
- The relationship between conventional LLMs, reasoning models, and agent harnesses
- On the benefits of learning through code implementations
- The scope of this video series
- Installing Python dependencies
- Choosing hardware and checking CUDA or Apple Silicon support
- Using a remote GPU with VS Code and troubleshooting the environment
Watch video 1 · Setup instructions
2. Loading a base model, text generation, and KV caching
Before adding reasoning methods in the upcoming videos, we need a working base model. Instead of training it from scratch, since that would be a whole other book (see Build A Large Language Model (From Scratch)), we start with a pre-trained model.
Here, we load the pre-trained weights of a Qwen3 0.6B model. Then, we implement the text-generation process via a streaming text generator and add KV caching and model compilation to make text generation more efficient.
These implementations give us the baseline and base components we will evaluate and improve in the following videos.
Topics covered
- Encoding text as tokens and decoding it back into text
- Downloading Qwen3 weights and loading the model on the selected compute device
- Going over tensor shapes, logits, and next-token predictions
- Implementing greedy decoding and streaming text generation
- Handling end-of-sequence tokens and measuring generation speed
- Adding a KV cache to reuse attention keys and values
- Combining KV caching with torch.compile and comparing hardware performance
Watch video 2 · Chapter 2 code
3. Building a verifier for evaluation and reinforcement learning
In this lecture, we build a math-answer verifier step by step, from answer extraction to normalization and checking mathematical equivalence. We then use it to evaluate the model on MATH-500.
We use this verifier to evaluate the base model’s performance on the MATH-500 evaluation dataset.
However, the verifier also has a second job later in the series: providing the reward signal for reinforcement learning with verifiable rewards (RLVR).
Topics covered
- Intro to verifiers and how they fit into LLM evaluation methods (other methods are covered in the book’s appendix)
- Reusing answer verification as a reward signal for RLVR (more on that in chapter 6)
- The verifier flow, including extracting final answers, normalizing answers and checking mathematical equivalence
- Building an answer grader and an evaluation loop for MATH-500
- Comparing base and reasoning models on MATH-500
Watch video 3 · Chapter 3 code
4. Inference-time scaling: sampling and self-consistency
Now we explore ways to improve answers while keeping the model weights fixed. But before implementing specific inference-scaling techniques, we first implement methods to generate diverse answers from the same prompt (this is standard practice in proprietary LLM chatbots, such as ChatGPT).
Concretely, we implement temperature scaling, multinomial sampling, and top-p filtering to generate diverse responses. Then we combine multiple responses through self-consistency, using majority voting over their final answers.
We also examine the tradeoff between answer accuracy and the extra computation needed to generate more candidates.
Topics covered
- Comparing how additional compute benefits model performance during training and during inference
- A gentle intro to inference-scaling through chain-of-thought prompts
- Implementing temperature scaling and multinomial sampling to generate diverse answers
- Adding top-p filtering to control the quality of the diverse answers
- Combining candidate answers through self-consistency, one of the most popular inference scaling techniques
- Evaluating MATH-500 accuracy against the cost of additional inference compute
Watch video 4 · Chapter 4 code
5. Inference-time scaling: scoring and self-refinement
Building on the previous video, we implement scoring functions to compare multiple answers generated from the same prompt. We can also use these scoring methods to break ties in self-consistency (covered in the previous lecture).
The scoring methods covered here include computing token and sequence log probabilities (this will also be relevant in the later reinforcement learning chapters). We then implement a self-refinement loop in which the model critiques an answer and attempts a revision.
The evaluation lets us check how useful those additional steps are for our model.
Topics covered
- Using scoring functions for comparing multiple answers and breaking ties
- Implementing a simple heuristic scorer
- Computing token probabilities and sequence likelihoods (also relevant for the following lectures)
- Using log probabilities for improved numerical stability
- Building a self-refinement loop for generating critiques and revised answers
- Evaluating self-refinement on MATH-500
Watch video 5 · Chapter 5 code
6. Reinforcement learning: implementing GRPO for RLVR
In this lecture, we move on to implementing a training technique called Reinforcement Learning with Verifiable Rewards (RLVR). We first implement a simplified version of Group Relative Policy Optimization (GRPO), generate groups of responses, score them with our verifier, and use their relative rewards for the model weight updates during training.
Topics covered
- Generating reasoning traces and understanding the relationship between answer lengths and problem-solving abilities
- Discussion and comparison of two important LLM post-training methods: Reinforcement Learning with Human Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR)
- Turning verifier scores into rewards and relative advantages
- Computing sequence log probabilities and a simplified GRPO loss
- Assembling the training loop with logging and checkpoint saves
- Evaluating checkpoints and checking training stability and memory needs
Watch video 6 · Chapter 6 code
7. Reinforcement learning: tracking, clippe
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.