后培训探索Demoscenes模式
关于演示文法和未完成代码的可能性
最近，我对世界的降级和大小编码感到着迷，最终我看到HellMood杰出的MS DOS演示"记忆". 在阅读他的写作时，我开始对作者如何到达其视觉效果感兴趣。 我试图创造任何我自己，所以我希望从中吸取一点灵感。
我注意到这些效果可以追溯到早期的实验，包括替代执行和从之前的演示中借用的一些想法。 我开始想知道，同样的基本想法可能还采取了什么其他方向。 我决定从几个现有的演示开始，最简单的演示，并按角色修改它们，观察这些改变如何影响视觉感受。
我碰到了Dwitter.net，这就像推特但140个角色JavaScript演示。 他们已经支持“remix ” —— 任何哑巴，这样我们就可以在它们之上更上一层楼。 顺便说一句，网站中最受欢迎的Dweet与来自Memories的 Parallax检查板效果相同！
我注意到一些重混体分享了相当一部分的源代码，与少数字符不同，而另一些则通过完全不同的应用实现了相似的视觉效果。
以下是一个例子，我从d/367547追溯到d/36555，而d/36467又由此推导出。 灰色下划线显示它们之间的共同点。 蓝色的显示在前两款之间是常见的，绿色的显示为下一对夫妇。
在语言学中，语言是富有成效的，即有限的词汇和有限的一套语法规则允许大量不同的表达。 编程语言也共享此属性。 让我们来考虑一下上述的Dweets。 它们包含相互重叠的下弦，但其结果不同。 假设你拿一个140个角色程序，并删除它的最后30个角色。 即使有一个只有64个字符的假设字母，也有几乎1.5×10^54可能的完成字符串可以形成。 几乎所有都会在战术上无效或产生无趣的结果。 尽管如此，可能性的数量是天文的。
人类不可能详尽地列举所有这些，然而，部分方案的继续并非同样合理。
由于一个部分程序，某些字符和表达方式比其他字符和表达方式更可能遵循。 这几乎就是自递性语言模型的功能。 我想知道我是否可以使用这样的模型来取出未完成的Dweets 的可信续作并想象每个程序可能采取的不同方向
然后，我可以外科切除下弦或混合和匹配与我自己的审美口味。
我对自己在开始演示后“完成”演示的能力感到相当沮丧
大约在同一时间，我看到Mithil Vakde的微博(你可能最近在HackerNews上见过他)
这给了我另一个实验的借口 - 我可以把自己的美学品味注入我的自动完成模型吗？这真是太棒了。
我知道没有人能写出一个决定演示是否美丽的功能，但人们可以通过比较例子来表达他们的喜好，对吗？
考虑到这一点，我决定让我的GitHub赞助商出钱租一个GPU，然后在训练后给自己的模特。
第一个障碍是找到合适的数据集。 我最初尝试用Claude Sonnet 5.5来生成合成的Dweets，但发现它们从美学上来说并不令人愉快。 幸运的是，Dwitter.net 提供了一个API，我曾经建造一个由25k Dweets组成的适中数据集，与再~1.3k作为验证集一起用于培训。
我用Qwen2.5-Code-7B作为基模。 它很难完成我测试过的大部分Dweets，这是可以理解的，考虑到他们的语法有多奇特。 Dweets广泛使用代码高尔夫技巧和数学短手，与代码相比，这些技巧和短手是相当不寻常的，是一般目的编码模型通常遇到的。
我决定继续尝试对尸体进行预训，看它是否有效，我测量了训练前和训练后个人代币的溢价。 超常是模型给一个符号分配的负对数概率，信物的可能性越小，对模型来说就越出乎意料。
在下图中，更深的红色表示更大的外溢。 在CPT之后，很多代码变得明显不足为奇(尤其是不熟悉的编程规范).
perplexity从28.63降至8.57，这意味着模型在预测这些微小的怪异程序的结构方面已经大大改进。
尽管产生的方案更加一致，但它们仍然缺乏某种“图”。 幸运的是，Dwitter对每个Dweet有“惊人”的分数。 我决定用它来构建课程
我以“优异”的分数排名前5克德威特，最初，每个被采样的dweet有同样的可能性。 超过400个训练步骤，我逐渐增加了分给排名较高的课程的分量，直到前1000名约占采样概率的67%。
这真是一次突破！产生的续作在视觉上更加有趣。
有趣的是，象征性可能性方面的改进甚至扩大到在课程培训期间从未进行过抽样的Dweets。 Perplexity从7.42跌至5.65，虽然收益分布在可口级相当均匀。
超过7000个Dweets，课程培训提高了所有优秀阶层的象征性可能性，包括培训期间从未抽取过的方案。 最初的模型发现很难，但对于方案来说，收益尤为明显。
我现在有一个模型可以产生相当漂亮的续作。 我在想，我是否可以把自己的美学偏好纳入其中，从而进一步推动这项工作。
9月中旬，我看到苏里亚的"训练AI用密码作画"一文，他们用RL来用p5js来制作水彩画。 他们试验了不同的奖励功能，然后用对等的比对来比对一批手额画。 我想起了这个想法，并想到尝试类似Demoscener(反复称它为“模型”并不合适)的尝试。
我选择使用GRPO来做RLHF，由我自己担任奖励功能。
每一个未完成的Dweet，我抽取了一组8个完成，把它们放在一起，选择了我发现最审美的。 如果没有人向我求情，我跳过了队伍。 我利用这些偏好来更新Demoscener的政策，我手动检查了4万部动画
我开始欣赏VLM和HPSv3的存在。 可惜已经太迟了
经过几轮训练，我想看看我的口味究竟在德摩斯克纳身上消退了多少。 我召集了一个相当不愿意的朋友，从几个检查站(我随机挑选了4个)判断400套演示，却不知道是哪个模式产生的。
结果显示4号检查站是最有利的所以我用它来推断我的Mac
现在到了边上探索的尽头，我建立了一个小界面，在那里，我可以产生几个续作从同一个前缀，使它们并肩并肩，并且从任何我的眼睛。 然后，我可以调整代码，产生更多的续作，并不断探索。
每个分支都代表着可能的延续，我可以用手来对比，并用作进一步探索的起点。
能够走过这些可能性是件很美妙的事。 我想我会花更多的时间这样做。
您可以在HuggingFace上找到Demoscener-7B和数据集。
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Post training a model to explore Demoscenes
On the grammar of demoscenes and the possibilities of unfinished code
Lately, I have been fascinated by the world of demoscening and sizecoding and eventually I came across HellMood’s remarkable MS DOS demo “Memories”. Whilst reading through his writeup, I became interested in how the author had arrived at some of its visual effects. I was struggling to create any of my own, so I hoped to squeeze out a little bit of inspiration out of that.
I noticed that these effects could be traced back to earlier experiments consisting of alternative implementations and several ideas borrowed from previous demos. I began wondering how many other directions the same underlying ideas might have taken. I decided to start with a few existing demos, the simplest ones of all, and modify them character by character to observe how those changes can affect the visual feel.
I came across Dwitter.net, which is like Twitter but for 140 character JavaScript demos. They already supported “remix”-ing any dweet so that we can build on top of them. Co incidentally, the most liked Dweet on the site shared the same effect as the parallax checkerboard effect from Memories!
Digging a little deeper, I noticed some remixes shared substantial portions of their source code, differing by only a few characters, whilst others achieved similar visual effects through entirely different implementations.
Here is an example, I traced back d/36555 from d/367547 which in turn is derived from d/36467. The gray underlined substrings show what is common across all of them. Blue shows what’s common in between the first two, and green shows for the next couple.
In linguistics, language is productive i.e. a finite vocabulary and a finite set of grammatical rules permit an enormous number of different expressions. Programming languages share this property too. Now let’s consider the aforementioned Dweets. They contain overlapping substrings, but their rendered results differ. Suppose you take a 140 character program, and remove its final 30 characters. Even with a hypothetical alphabet of just 64 character, there are almost 1.5 x 10^54 possible completion strings that can be formed. Almost all will be syntactically invalid or produce uninteresting results. Nevertheless, the number of possibilities is astronomical.
It’s not humanly possible to enumerate all of them exhaustively and it would be foolish to to generate and test all possible strings. However, the continuations of a partial program are not equally plausible.
Given a partial program, certain characters and expressions are considerably more likely to follow than others. This is pretty much how autoregressive language models work. I wondered whether I could use such a model to sample plausible continuations of unfinished Dweets and visualize the different directions each program could have taken.
And then, I could surgically delete substrings or mix-and-match them to my own aesthetic taste.
I had also grown rather frustrated with my own ability to “finish” a demo after starting one, so having an autocomplete that understood the peculiar language of Dweeting rather looked like an appealing prospect to me.
Around the same time, I came across Mithil Vakde’s tweet (you might have seen him on HackerNews recently).
That gave me another excuse to experiment - could I inject my own aesthetic taste as a prior into my autocomplete model? That’d be really amazing.
I understand that no one can write a function that determines whether a demo is beautiful, but people can express their preferences by comparing examples, right?
With that in mind, I decided to put my GitHub Sponsors payout towards renting a GPU and post training a model of my own.
The first hurdle was finding a suitable dataset. I initially experimented with generating synthetic Dweets using Claude Sonnet 5.5 but found them not at all aesthetically pleasant. Fortunately, Dwitter.net provides an API, which I used to build a modest dataset of 25k Dweets to be used for training along with ~1.3k more as a validation set.
I used Qwen2.5-Code-7B as the base model. It struggled to complete most of the Dweets I tested it on, which was understandable given how peculiar their syntax is. Dweets make extensive use of code golfing tricks and mathematical shorthand, which are rather unusual compared with the code a general purpose coding model typically encounters.
I decided to try continued pretraining (CPT) on the corpus, hoping to shift the model’s probability distribution towards the pecular language of dweeting. To see whether it had worked, I measured the surprisal of individual tokens before and after training. Surprisal is simply the negative log probability a model assigns to a token given everything preceding it. The less likely a token is, the more surprising it is to the model.
In the figure below, darker red indicates greater surprisal. After CPT, much of the code becomes noticeably less surprising (particularly the unfamiliar programming idioms).
Across 30 held out dweets, perplexity fell from 28.63 to 8.57, which means that the model had become substantially better at predicting the structure of these tiny weird programs.
Although the generated programs were more consistent, they still lacked a certain “charm”. Fortunately, Dwitter has an “awesomeness” score for every Dweet. I decided to use it to construct a curriculum.
I ranked the top 5k dweets by their “awesomeness” scores and constructed a curriculum that gradually favoured the more appreciated programs. Initially, every dweet had an equal probability of being sampled. Over 400 training steps, I progressively increased the weight assigned to higher ranked programs, until the top 1000 accounted for roughly 67% of the sampling probability.
This turned out to be a breakthrough! The generated continuations became much more visually interesting.
Interestingly, the improvements in token likelihood extended even to Dweets that had never been sampled during curriculum training. Perplexity fell from 7.42 to 5.65, although the gains were spread fairly evenly across awesomeness ranks.
Across 7000 Dweets, curriculum training improved token likelihood across all awesomeness ranks, including programs never sampled during training. The gains were particularly pronounced for programs the original model found difficult.
I now had a model that could produce rather beautiful continuations. I wondered whether I could take this further by incorporating my own aesthetic preferences.
In mid-September, I came across Surya’s article on Training AI to Paint with Code, where they had used RL to produce watercolor paintings using p5js. They experimented with different reward functions before settling on pairwise comparsions against a collection of hand rated paintings. I remembered that idea and thought of trying something similar with Demoscener (it doesn’t feel alright to call it “the model” repeatedly).
I opted to do RLHF using GRPO, with myself acting as the reward function.
For each unfinished Dweet, I sampled a group of eight completions, rendered them side by side and selected the one I found most aesthetically pleasing. If none appealed to me, I skipped the group. I used these preferences to update Demoscener’s policy, then sampled fresh continuations and repeated the process over roughly 9–10 rounds of 500 groups each. Altogether, I manually inspected around 40,000 animations.
Somewhere around the 10,000th animation, I began to appreciate the existence of VLMs and HPSv3. Sadly, it was too late.
After several rounds of training, I wanted to see how much of my taste had actually rubbed off on Demoscener. I enlisted a rather unwilling friend to judge 400 sets of demos from a few of the checkpoints (I picked 4 randomly), without knowing which model had produced which.
The results showed that checkpoint 4 was the most favorable, so I used that to run inference on my Mac.
Now coming to the end of the side quest, I built a little interface where I could generate several continuations from the same prefix, render them side by side and branch off from whichever one caught my eye. I could then tweak the code, generate more continuations, and keep exploring.
Each branch represents a possible continuation, which I could compare visually, tweak by hand, and use as the starting point for further exploration.
There is something quite lovely about being able to wander through these possibilities. I think I'll be spending a little more time doing just that.
You can find Demoscener-7B and the dataset on HuggingFace.
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.