对一个GPU进行批量分类和决定推论。
spinf-serve读取了一段内容一次，并在同道中解答了许多关于它的问题。 每个答案都是你给出的选项的概率，而不是生成的文本。 它运行于一款24GB NVIDIA GPU上并服务于OpenAI相容的API，与主机spinf服务相同的API. 阿帕奇2.0 (英语).
我们为自己的工作建造了这个引擎：每天的市场指数从数百万篇新闻文章中读出，每篇文章都问了几十个狭义，校正的问题。 为聊天而建造的一款服务引擎重新读取每个问题的内容，一次生成一个符号，这使得这项工作缓慢而昂贵。 spinf-serve将每个问题都排出一个读取内容的分支，并将许多内容打入一个前传。 分出分出分出是始发地和最强地。
同样的机制是现在所谓的决定模型：一个模型，它通过一组固定的答案返回概率(是或否，标签，级别)而不是一段落。 任何开放的指令模型都以这种方式在spinf-serve中工作，专用的决策模型按原样运行。 主机spinf API是从这个引擎中发展出来的；我们正在出版它，因为人们要求用自己的硬件运行它。
这不是聊天服务器。 除非您要求，否则任何事物都无法产生，它是为了吞吐量而建造的，而不是为了单一请求的暂时性而建造的。
Linux,NVIDIA GPU配有24个GB和fp8的拉力核心(Ada,Hopper或Blackwell:RTX 4090 / 5090,L4,L40S,RTX 6000 Ada,H100)，是支持CUDA 13,Python 3.12的驱动程序。 安佩尔牌以倒数8克重来跑取(配置).
pip 安装 “ spinf- serve [server] ”
spinf-serve service sinf/gemma-4-12b-fp8# 首发时下载检查点；服务于港口 8000
或图像，安装了 NVIDIA 集装箱工具包(容量保存模型并编译缓存);
图像名称后面的参数是spinf-serve serve serve:
docker 运行 -- gpus 全部 -p 8000: 8000 -v spinf-cache:/root/.cache ghcr.io/spinfinc/spinf-serve
便问。 两种分类和一种取自单一读取的内容：
校对：Soup
-H"Content-Type：应用程序/json"\?
-d '{
"模型":"平克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克
"灵枢": [
{"作用":"用户".
"内容":"客户支持信息：\n I被收取了两次9月的费用，请退还其中一项费用" }
],
"评分":{
"泻":[
{"id":"论".
"通":"\ntopic (billing, technical, account, limiting，或取消):{?}"
"选项": ["计费","技术","账户","航运","注销"]}
{"id":"还".
"通":"客所求还？曰：？曰：?
"可选": ["是","否"]
{"id":"月"作"月".
"通鉴":"\：取出一月取取出二分？曰："取出一月取出一分，取出一分。
]
}
}'
对于每个得分问题，答案给出了每个选项的概率，对于空内容(从12B模型得到的实际数字)的概率相同：
{"id":"退款","合并":[{"价值":}"可选":
{"文本":"是","日志":-0.90,"p":0.72,"地板 p":0.60}
{"文字":"否","日志":-1.86,"p":0.28,"地板":0.40}]
"余":"0.44}}] }.
p 是您选项中的概率。 地板 p是同一个问题，没有内容，所以p-地板 p是什么
添加的内容(这里的题目是“计费”从0.47移到0.91)。 剩余是用到文字上的概率
在你的选择之外。 对于月份，答复列出了最可能的答案及其概率("九月"第一).
同一机身有 API 键，运行于 api.spinf.com; OpenAI SDKs 工作通过 /v1/chat/ 完成
附加体中的分数块(快速启动)。
- 一个读，许多问题。 内容读取一次；每个问题都是同一个通行证的分支，只花费自己的代币。 占位符将一个模板变成一个问题网格。
- 得分数，你可以门槛。 每个选项的概率及其下限和日志概率；决定是您在自己的数据(校正)上选择的阈值。
- 从同一案文中提取的数值为：正文、{n}数字和{$}按其概率解码的数额
或生成到槽中的免费文本。
- 同样的分数无论其他的在批。 有了fp8检查站，一个内容的分数是比特-同
独自一人或与数以千计的其他人打包，跨越电话、天数和服务器。 二时的系统指印称
答复不具有可比性。
- 内容中的图像，音频和视频帧，用于读取它们的模型。
- 快速包。 一项工作(内容节制、分类、产品上市、评论、品牌、娱乐)的准备、校正问题集，预先装入并在请求中取名(以下)。
- 主办或自主办。 客户端在此服务器和 api. spinf.com 之间切换，方法是更改基址和
钥匙互通客户谈话；
agent技巧教给一个编码agent的API，这个服务器和包格式。
在一款RTX 4090上用Gemma 4 12B在fp8中测量：24个维基新闻文章，约470个令牌，56个双选题
每一个，总共1,344个问题(固定装置/设置的公共工作)，通过HTTP发送到spinf-serve和vLLM 0.26
前缀缓存。 剧本和完整结果以基准测试/为单位。
vLLM 已经通过前缀缓存重用每篇文章的56个问题。 它不能做的是用一个前传来回答他们：它运行着1,344个单托请求，而spinf-serve将56个问题作为一个读取的分支来放出，并将24篇文章打入两个16k-tok的口。 差距随着每个内容的提问次数而增加。
实际上，每4090个12B型：自旋/调试包(每条电文106个问题，22个类别)
每小时得分约2万条；一篇新闻文章问了74个问题，每秒读取近700个问题。
提取槽增加了解码工作(搜索槽的费用高达200至600个记分符，生成)
大约两千)，但分享阅读。
一个工作很少是一个问题：内容温和度是106个，每个都与正常流量相适应，并结合
进入类分和决定。 一个即时包就是一个持有问题的文件，内容是如何呈现的
对模型，答案是如何变成分数的以及分数是如何变成决定的。 服务器预装包和
请求名称 1 ("packs": [{"id": "spinf/ modelation" }])，获取包的类别和下个决定
从同一字读作
官方的包以回旋为快取，默认加载。 服务器
从任何 GitHub 寄存器或本地文件夹中装入包，并带有 -- packs; PACKS. md 列表
包我们知道，我们的和别人的，一个问题
新增你的，我给你的。 格式，请求字段和来源：docs/即时包.md.
每个都是一个检查站，我们量化和校准，发布在Hugging Face枢纽上
其配方(数量化/)；引擎支撑着它经过对等测试的修订。 任何其他中央检查站
一个被支持的建筑家族按其路径运行，在负载时间被量化。
docs/：快速起动，实例，评分API，配置，写出问题，校正，即时包，FAQ；指向(决定模型，你能够采取行动的概率，几发分级，情绪，勘测编码，节制，意向路由)；以及关于引擎如何工作的工程文件(architecture，批量变化，精度，Qwen堆，提取).
第0.1版 (中文(简体) ). 发动机(得分，取出，生成，介质，批量变异，两个模型堆栈)是完整的，对所记录的参考文献进行等同测试；服务器会讲主机服务的公共API；四个检查站是公开的。 文件/README.md中列出了仍然缺失的内容。
问题、想法和错误都涉及这个寄存器的问题。 没有论坛或聊天，没有什么太小。
spinf serve/ 持有引擎(protocol，编译器，打包器，计分器，提取器，服务)，服务型号'.
列表，包模块(纯Python，客户端和主机API也使用)，每个架构一个模型堆栈
型号/ 中的家族，型号/ 中的量化方案，服务器/ 中的 HTTP 服务器和 spinf- serve 命令。
量化/ 每个公布的检查点有一个文件夹(检索、校正输出、型号卡)、固定/公众
等同固定和参考，基准测试/吞吐量比较。 代码地图是
医生/教师
uv 同步 -- 外部服务器
uv 运行 pytest # CPU 测试(官方包的黄金固定装置需要。/spinf-prompts 或 SPINF PROMPTS DIR)
uv 运行 ruff 检查。 uv 运行 ruff 格式 - 检查。
CUDA VISIBLE DEVICES=0 uv 运行自旋-serve 等同 -- -- 固定装置/wikinews 20261024.json.gz \
--参考定着/wikinews 20261024. spinf-gemma-4-12b-fp8.json.gz --passes 2
触及一个内核，一个编译边界或以GPU等分的量化计划船运行的每个变化：每个已公布的检查点(fixtures/)记录的参考文献上的位同日志概率。 依赖性是完全固定的；它们只随着对等运行而改变。 见捐款.md。
Apache 2.0 (英语：LICENSE, Notice). 我们运行和重新量化的模型，我们开发的项目，我们的技术
数据记入ACKNOWLEDGEMENS.md。
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Bulk classification and decision inference on one GPU.
spinf-serve reads a piece of content once and answers many questions about it in the same pass. Each answer is a probability over the options you give, not generated text. It runs on one 24 GB NVIDIA GPU and serves an OpenAI-compatible API, the same API as the hosted spinf service. Apache 2.0.
We built this engine for our own work: a daily market index read from millions of news articles, each article asked dozens of narrow, calibrated questions. A serving engine built for chat re-reads the content for every question and generates one token at a time, which made that job slow and expensive. spinf-serve lays every question out as a branch of one read of the content and packs many contents into one forward pass. Bulk classification is where it started and where it is strongest.
The same mechanism is what is now called a decision model: a model that returns a probability over a fixed set of answers (yes or no, a label, a level) instead of a paragraph. Any open instruction model works this way in spinf-serve, and dedicated decision models run as they are. The hosted spinf API grew out of this engine; we are publishing it because people asked to run it on their own hardware.
It is not a chat server. Nothing is generated unless you ask for it, and it is built for throughput rather than for the latency of a single request.
Linux, an NVIDIA GPU with 24 GB and fp8 tensor cores (Ada, Hopper or Blackwell: RTX 4090 / 5090, L4, L40S, RTX 6000 Ada, H100), a driver supporting CUDA 13, Python 3.12. Ampere cards run with int8 weights instead (configuration).
pip install "spinf-serve[server]"
spinf-serve serve spinf/gemma-4-12b-fp8 # downloads the checkpoint on first start; serves on port 8000
Or the image, with the NVIDIA Container Toolkit installed (the volume keeps the model and the compile caches;
arguments after the image name are those of spinf-serve serve):
docker run --gpus all -p 8000:8000 -v spinf-cache:/root/.cache ghcr.io/spinfinc/spinf-serve
Then ask. Two classifications and one extraction from a single read of the content:
curl http://localhost:8000/v1/score \
-H "Content-Type: application/json" \
-d '{
"model": "spinf/gemma-4-12b-fp8",
"messages": [
{ "role": "user",
"content": "Customer support message:\nI was charged twice for September, please refund one of the charges." }
],
"scoring": {
"queries": [
{ "id": "topic",
"template": "\n\nTopic (billing, technical, account, shipping, or cancellation):{?}",
"options": [" billing", " technical", " account", " shipping", " cancellation"] },
{ "id": "refund",
"template": "\n\nIs the customer asking for a refund?\nAnswer:{?}",
"options": [" yes", " no"] },
{ "id": "month",
"template": "\n\nQ: Which month was charged twice?\nA: \"{*}\"" }
]
}
}'
For each scored question the response gives every option's probability, and the same probability on empty content (real numbers from the 12B model):
{ "id": "refund", "combinations": [ { "values": {}, "options": [
{ "text": " yes", "logprob": -0.90, "p": 0.72, "floor_p": 0.60 },
{ "text": " no", "logprob": -1.86, "p": 0.28, "floor_p": 0.40 } ],
"residual": 0.44 } ] }
p is the probability among your options. floor_p is the same question with no content, so p - floor_p is what
the content added (here topic moves " billing" from 0.47 to 0.91). residual is the probability that went to words
outside your options. For month the response lists the most likely answers with their probabilities ("September" first).
The same body, with an API key, runs against api.spinf.com; the OpenAI SDKs work through /v1/chat/completions with
the scoring block in extra_body (quickstart).
- One read, many questions. The content is read once; every question is a branch of the same pass and costs its own tokens only. Placeholders turn one template into a grid of questions.
- Scores you can threshold. A probability per option with its floor and log-probability; a decision is a threshold you choose on your own data (calibration).
- Extraction from the same read: {*} text, {n} numbers and {$} amounts decoded with their probabilities,
or free text generated into a slot.
- The same score whatever else is in the batch. With the fp8 checkpoints, a content's scores are bit-identical
alone or packed with thousands of others, across calls, days and servers. The system_fingerprint says when two
responses are not comparable.
- Images, audio and video frames in the content, for the models that read them.
- Prompt packs. Ready-made, calibrated question sets for a job (content moderation, triage, product listings, reviews, brand, entertainment), preloaded and named in a request (below).
- Hosted or self-hosted. A client switches between this server and api.spinf.com by changing the base URL and
the key. spinf-client talks to both; the
agent skills teach a coding agent the API, this server and the pack format.
Measured on one RTX 4090 with Gemma 4 12B in fp8: 24 Wikinews articles of about 470 tokens, 56 two-option questions
each, 1,344 questions in all (the public job set in fixtures/), sent over HTTP to spinf-serve and to vLLM 0.26 with
prefix caching. The script and the full results are in benchmarks/.
vLLM already reuses each article across its 56 questions through the prefix cache. What it cannot do is answer them in one forward pass: it runs 1,344 single-token requests, while spinf-serve lays the 56 questions out as branches of one read and packs the 24 articles into two 16k-token passes. The gap grows with the number of questions per content.
In practice, per 4090 with the 12B model: the spinf/moderation pack (106 questions per message, 22 categories)
scores about 20,000 messages per hour; a news article asked 74 questions is read at nearly 700 questions per second.
Extraction slots add decoding work (a searched slot costs about as much as 200 to 600 tokens of scoring, a generated
one up to about 2,000) but share the read.
A job is rarely one question: content moderation is 106 of them, each calibrated against normal traffic and combined
into category scores and a decision. A prompt pack is one file holding the questions, how the content is presented
to the model, how the answers turn into scores and how the scores turn into a decision. The server preloads packs and
a request names one ("packs": [{"id": "spinf/moderation"}]), getting the pack's categories and decision back next
to its own questions, from the same read.
The official packs are in spinf-prompts, loaded by default. The server
loads packs from any GitHub repository or local folder given with --packs; PACKS.md lists every
pack we know of, ours and others', and an issue
adds yours. Format, request fields and sources: docs/prompt-packs.md.
Each is a checkpoint we quantized and calibrated, published on the Hugging Face Hub
with its recipe (quantizations/); the engine pins the revision it was parity-tested on. Any other Hub checkpoint of
a supported architecture family runs by its path, quantized at load time.
docs/: quickstart, examples, scoring API, configuration, writing questions, calibration, prompt packs, FAQ; the guides (decision models, probabilities you can act on, few-shot classification, sentiment, survey coding, moderation, intent routing); and the engineering documents on how the engine works (architecture, batch invariance, precision, the Qwen stack, extraction).
Version 0.1. The engine (scoring, extraction, generation, media, batch invariance, the two model stacks) is complete and parity-tested against recorded references; the server speaks the public API of the hosted service; the four checkpoints are public. What is still missing is listed in docs/README.md.
Questions, ideas and bugs go to the issues of this repository. There is no forum or chat, and nothing is too small.
spinf_serve/ holds the engine (protocol, compiler, packer, scorer, extract, service), the served models'
manifests, the packs module (pure Python, also used by the client and the hosted API), one model stack per architecture
family in models/, the quantization schemes in quant/, the HTTP server in server/ and the spinf-serve command.
quantizations/ has one folder per published checkpoint (recipe, calibration output, model card), fixtures/ the public
parity fixture and references, benchmarks/ the throughput comparison. The map of the code is
docs/ARCHITECTURE.md.
uv sync --extra server
uv run pytest # CPU tests (the official packs' golden fixtures need ../spinf-prompts or SPINF_PROMPTS_DIR)
uv run ruff check . && uv run ruff format --check .
CUDA_VISIBLE_DEVICES=0 uv run spinf-serve parity --fixture fixtures/wikinews_20261024.json.gz \
--reference fixtures/wikinews_20261024.spinf-gemma-4-12b-fp8.json.gz --passes 2
Every change that touches a kernel, a compile boundary or a quantization scheme ships with the GPU parity run: bit-identical log-probabilities on the recorded reference of each published checkpoint (fixtures/). Dependencies are pinned exactly; they change only together with a parity run. See CONTRIBUTING.md.
Apache 2.0 (LICENSE, NOTICE). The models we run and re-quantize, the projects we build on, the techniques we
implement and the data we checked on are credited in ACKNOWLEDGEMENTS.md.
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.