Kyutai CADRE：剪接世界模式 Lag 的视频自动编码器 230 ms以下

中文版正在随流水线自动整理，稍后刷新即可；可先阅读下方英文原文。

本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

Kyutai has released CADRE, open causal diffusion representation autoencoders for interactive world models. The Kyutai CADRE video autoencoder targets one problem: action lag. Many world models borrow autoencoders from text-to-video generators. Those autoencoders pack many frames into one latent, so every action waits. Per the research paper, CADRE world models run at 20 fps on 1 H100 and react within 230 ms.

TL;DR

- Size: 7 autoencoders, each about 476M trained parameters plus a frozen 86M DINOv3 ViT-B/16. The world model is 1.2B parameters. Video runs at 288×512, 20 fps.

- Runs on: 1 H100 (bf16, batch 1). Quantized variants: Not disclosed.

- Performance: CADRE forms most of the latency-quality frontier. It is as fast and better in 42 of 48 comparisons with MIRA, Wan 2.2, Cosmos and LTX-2.

- Best: Driving FVD of 162.6 (CADRE-2×16×16, 1 step) against 599.7 for Wan 2.2.

- Worst: Robot manipulation action following (ARR) of 0.45 for CADRE-2×32×32, against 0.85 for Wan 2.2.

What is CADRE?

CADRE is a family of video autoencoders from Kyutai built for action-conditioned world models. A world model generates in the autoencoder’s latent space. So the autoencoder’s compression sets both cost per frame and action delay.

Compression is written τ×s×s. τ is frames per latent, and s is the spatial factor. The research gives the latency as τ/f plus generation and decode time. At τ = 8 and 20 fps, the chunk alone adds 400 ms. Humans notice their own movement lagging beyond about 230 ms.

How does CADRE work?

The encoder reuses the MIRA recipe. A frozen DINOv3 extracts per-patch features from each frame. One trained 3D convolution then compresses them into a 64-channel latent. Only this bottleneck changes between the 7 members.

The decoder is a causal diffusion transformer with 24 blocks. It streams chunk by chunk. Progressive self-distillation teaches it to decode in 1 or 2 steps. This happens in a single training stage, without an adversarial loss.

The autoencoders train on 23 public video sources, about 76k hours. The mix spans web, robot, driving and game video.

How was CADRE evaluated?

Kyutai trained the same 1.2B causal diffusion transformer on each autoencoder. It used 4 environments: robot manipulation (ABC), human manipulation (EgoDex), driving (PhysicalAI AV) and indoor navigation (RealEstate10K). Each model sees 2 s of real video, then generates 4.8 s.

Scores cover image quality (FID), motion quality (FVD) and action following (ARR). ARR uses a frozen inverse dynamics probe. An ARR of 1 matches real video.

What do the benchmarks show?

On one H100, CADRE-2×32×32 reacts in 185 ms and sustains 23.4 fps with 1 decoding step. The Wan 2.2 autoencoder needs 317 ms, LTX-2 492 ms and Cosmos 605 ms. DC-AE is faster at 118 ms but frame-by-frame.

The gap is widest with a moving camera. In driving, Wan 2.2 FVD reaches 599.7 and LTX-2 913.3. In indoor navigation, LTX-2 ARR falls to 0.04. Against the retrained MIRA decoder, the 1-step diffusion decoder lowers FVD by 1.4 to 2.5×.

What are the 4 findings?

- Decode by diffusion: at strong compression, perceptual regression blurs, while diffusion samples 1 sharp video.

- Compress time in the autoencoder: patching inside the world model loses detail. At 8 frames, ARR drops from 0.72 to 0.56.

- Camera motion decides compression: still cameras tolerate temporal compression. Moving cameras favor fewer frames and coarser spatial tokens.

- Coarse tokens hurt motion first: in robot manipulation, 32× spatial compression multiplies FVD by 2.4 over 16×. FID barely moves.

Comparison: CADRE vs other video autoencoders

Latency, fpsA, FVD, ARR and PSNR: CADRE paper, Tables 4, 6 and 7. CADRE uses 1 decoding step. All figures are self-reported by Kyutai.

Where does CADRE fall short?

CADRE does not win everywhere. In robot manipulation, Cosmos posts an FVD of 39.5 against 47.6 for the best 1-step CADRE. At the fast end, DC-AE beats CADRE-1×48×48 in manipulation. Wan 2.2 also reconstructs held-out video far better, at 35.02 dB PSNR.

Kyutai team argues reconstruction does not predict rollouts. Wan 2.2 ranks last on indoor rollout FVD.

Long latents leave visible seams between chunks. At τ = 8, boundary excess reaches 1.6 to 1.8 across environments. CADRE-4×24×24 is an outlier in navigation, with FVD 403 in driving. Each world model covers one environment at 288×512. Whether findings transfer to generalist models remains open.

How can developers use CADRE?

The code is MIT licensed. Weights on Hugging Face are CC BY-NC 4.0, so commercial use is excluded. The release covers 7 autoencoders, 4 longer-trained multi-resolution versions and 4 world models on CADRE-2×16×16. A live demo lets you drive them by keyboard.

Key Takeaways

- 7 open autoencoders, one bottleneck apart, isolate compression as the only variable.

- World models on CADRE run at 20 fps on 1 H100, under 230 ms lag.

- Driving FVD drops to 162.6, against 599.7 for Wan 2.2.

- Still cameras favor temporal compression; moving cameras favor spatial compression.

- Weights are non-commercial (CC BY-NC 4.0); code is MIT.

Check out the paper, the Kyutai blog, the GitHub repo and the model collection. All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us

Michal Sutter is a data science professional with a Master of Science in Data Science from the University of Padova. With a solid foundation in statistical analysis, machine learning, and data engineering, Michal excels at transforming complex datasets into actionable insights.

Auto-fetched and translated by AI Gem Box. All rights belong to the original author.