v1.0.0

Dependencies

flash-attention-2 and xformers are no longer needed. They are only optionally imported as reference implementations. The incompatibility between formatron and pydantic is handled with monkey-patching so the latter no longer needs to be pinned to an outdated release.

Decode / Prefill

Updated coop GEMM kernels and new INT8 GEMV kernels (supports mul1 codebook only) significantly improve throughput on Ampere and Blackwell, both of which have high bandwidth given their comparatively limited ALU throughput. Improvements on Ada are more modest since the memory and ALU pipes are better balanced to begin with.

C++ and CUDA graph paths are added for all models. Along with numerous other optimizations, these eliminate the CPU bottleneck for small models on the test setup (Threadripper 7960X), so speedups for those are disproportionate and will vary from system to system.

New MoE scheduler improves throughput on sparse prefill, but this is mostly felt on larger models. Only single-GPU test runs are included in the table below.

Prefill measured with a 4K-token prompt; decode is from an empty context. Results in tokens/second, greedy decoding. Select a version heading to compare the other columns against it.

Decode tokens/s

Model / bitrate
Qwen 3.5 0.8B4.00 bpw 268 432+61% 444+66%
Qwen 3.5 35B-A3B4.50 bpw 110 127+15% 140+27%
Qwen 3.5 9B2.50 bpw 85 101+19% 139+64%
Qwen 3.5 9B4.00 bpw 89 105+18% 123+38%
Qwen 3.6 27B3.00 bpw 29 35+21% 50+72%
Qwen 3.6 27B4.00 bpw 31 37+19% 44+42%
Llama 3.2 1B4.00 bpw 356 434+22% 473+33%
Llama 3.1 8B2.00 bpw 94 110+17% 146+55%
Llama 3.1 8B3.00 bpw 89 103+16% 134+51%
Llama 3.1 8B4.00 bpw 94 107+14% 126+34%
Gemma 4 12B3.50 bpw 55 65+18% 84+53%
Gemma 4 31B3.00 bpw 24 29+21% 40+67%
Trinity Nano 6B-A1B4.00 bpw 118 138+17% 138+17%

Prefill tokens/s, 4K

Model / bitrate
Qwen 3.5 0.8B4.00 bpw 29,766 30,844+4% 30,895+4%
Qwen 3.5 35B-A3B4.50 bpw 4,572 4,790+5% 4,802+5%
Qwen 3.5 9B2.50 bpw 3,907 3,960+1% 3,950+1%
Qwen 3.5 9B4.00 bpw 3,896 3,952+1% 3,934+1%
Qwen 3.6 27B3.00 bpw 1,166 1,173+1% 1,171
Qwen 3.6 27B4.00 bpw 1,165 1,171+1% 1,170
Llama 3.2 1B4.00 bpw 20,208 22,651+12% 22,586+12%
Llama 3.1 8B2.00 bpw 3,597 3,876+8% 3,875+8%
Llama 3.1 8B3.00 bpw 3,590 3,868+8% 3,863+8%
Llama 3.1 8B4.00 bpw 3,588 3,866+8% 3,861+8%
Gemma 4 12B3.50 bpw 2,363 2,412+2% 2,404+2%
Gemma 4 31B3.00 bpw 947 966+2% 967+2%
Trinity Nano 6B-A1B4.00 bpw 8,790 9,467+8% 9,364+7%

Decode tokens/s

Model / bitrate
Qwen 3.5 0.8B4.00 bpw 275 541+97% 532+93%
Qwen 3.5 35B-A3B4.50 bpw 141 168+19% 171+21%
Qwen 3.5 9B2.50 bpw 145 170+17% 185+28%
Qwen 3.5 9B4.00 bpw 135 152+13% 153+13%
Qwen 3.6 27B3.00 bpw 57 60+5% 63+11%
Qwen 3.6 27B4.00 bpw 51 54+6% 55+8%
Llama 3.2 1B4.00 bpw 503 562+12% 560+11%
Llama 3.1 8B2.00 bpw 169 189+12% 216+28%
Llama 3.1 8B3.00 bpw 164 177+8% 175+7%
Llama 3.1 8B4.00 bpw 150 160+7% 156+4%
Gemma 4 12B3.50 bpw 90 104+16% 103+14%
Gemma 4 31B3.00 bpw 50 51+2% 54+8%
Trinity Nano 6B-A1B4.00 bpw 144 172+19% 167+16%

Prefill tokens/s, 4K

Model / bitrate
Qwen 3.5 0.8B4.00 bpw 55,693 57,422+3% 57,389+3%
Qwen 3.5 35B-A3B4.50 bpw 8,993 9,307+3% 9,390+4%
Qwen 3.5 9B2.50 bpw 7,869 7,949+1% 7,916+1%
Qwen 3.5 9B4.00 bpw 7,847 7,847 7,853
Qwen 3.6 27B3.00 bpw 2,446 2,460+1% 2,459+1%
Qwen 3.6 27B4.00 bpw 2,441 2,456+1% 2,460+1%
Llama 3.2 1B4.00 bpw 41,169 46,216+12% 46,321+13%
Llama 3.1 8B2.00 bpw 7,476 8,117+9% 8,122+9%
Llama 3.1 8B3.00 bpw 7,491 8,138+9% 8,113+8%
Llama 3.1 8B4.00 bpw 7,470 8,116+9% 8,077+8%
Gemma 4 12B3.50 bpw 5,211 5,275+1% 5,259+1%
Gemma 4 31B3.00 bpw 2,042 2,082+2% 2,080+2%
Trinity Nano 6B-A1B4.00 bpw 16,296 17,041+5% 16,775+3%

Decode tokens/s

Model / bitrate
Qwen 3.5 0.8B4.00 bpw 261 503+93% 545+109%
Qwen 3.5 35B-A3B4.50 bpw 170 172+1% 188+11%
Qwen 3.5 9B2.50 bpw 170 184+8% 234+38%
Qwen 3.5 9B4.00 bpw 174 188+8% 204+17%
Qwen 3.6 27B3.00 bpw 67 74+10% 86+28%
Qwen 3.6 27B4.00 bpw 69 73+6% 79+14%
Llama 3.2 1B4.00 bpw 536 562+5% 623+16%
Llama 3.1 8B2.00 bpw 191 196+3% 259+36%
Llama 3.1 8B3.00 bpw 184 188+2% 222+21%
Llama 3.1 8B4.00 bpw 188 194+3% 209+11%
Gemma 4 12B3.50 bpw 99 118+19% 134+35%
Gemma 4 31B3.00 bpw 60 63+5% 73+22%
Trinity Nano 6B-A1B4.00 bpw 140 159+14% 158+13%

Prefill tokens/s, 4K

Model / bitrate
Qwen 3.5 0.8B4.00 bpw 73,791 76,537+4% 78,556+6%
Qwen 3.5 35B-A3B4.50 bpw 11,037 12,338+12% 12,613+14%
Qwen 3.5 9B2.50 bpw 11,058 11,208+1% 11,169+1%
Qwen 3.5 9B4.00 bpw 11,067 11,040 11,066
Qwen 3.6 27B3.00 bpw 3,377 3,398+1% 3,394+1%
Qwen 3.6 27B4.00 bpw 3,368 3,399+1% 3,389+1%
Llama 3.2 1B4.00 bpw 59,027 59,391+1% 59,333+1%
Llama 3.1 8B2.00 bpw 10,360 10,760+4% 10,675+3%
Llama 3.1 8B3.00 bpw 10,334 10,734+4% 10,691+3%
Llama 3.1 8B4.00 bpw 10,291 10,731+4% 10,671+4%
Gemma 4 12B3.50 bpw 7,053 7,072 7,109+1%
Gemma 4 31B3.00 bpw 2,781 2,809+1% 2,807+1%
Trinity Nano 6B-A1B4.00 bpw 18,462 20,102+9% 19,593+6%

Online cache quantization

New attention kernel adds online cache quantization, removing the need to stage temporary FP16 cache tensors for attention steps. Consequently, there is no longer a latency cost for cache quantization. For KV-heavy models, quantization often increases throughput instead.

Llama 3.1 8B 4.00 bpw

Qwen 3.6 27B 3.00 bpw

Llama 3.1 8B 4.00 bpw

Qwen 3.6 27B 3.00 bpw

Llama 3.1 8B 4.00 bpw

Qwen 3.6 27B 3.00 bpw

Architectures

Most models now have tensor-parallel modes, including Gemma 4. With sink support in the new attention kernel, GPT-OSS has been added. Added Mamba2 module enables NemotronH support.

HF architecture Model Multimodal TP
ArceeForCausalLM AFM ✓
AfmoeForCausalLM AfMoE
ApertursForCausalLM Apertus ✓
CohereForCausalLM Command-R etc. ✓
Cohere2ForCausalLM Command-A, Command-R+ etc. ✓
DeciLMForCausalLM DeciLM, Nemotron ✓
Dots1ForCausalLM dots.llm1 ✓
Ernie4_5_ForCausalLMErnie4_5_MoeForCausalLM ERNIE 4.5 ✓
Exaone4ForCausalLM EXAONE 4.0 ✓
Gemma2ForCausalLM Gemma 2 ✓
Gemma3ForCausalLMGemma3ForConditionalGeneration Gemma 3 ✓ ✓
Gemma4ForConditionalGenerationGemma4UnifiedForConditionalGeneration Gemma 4E2B/E4B currently not supported ✓ ✓
Glm4ForCausalLMGlm4MoeForCausalLM GLM 4, GLM 4.5, GLM 4.5-Air, GLM 4.6 ✓
Glm4vForConditionalGenerationGlm4vMoeForConditionalGeneration GLM 4.1V, GLM 4.5V ✓ ✓
GptOssForCausalLM GPT-OSSNEW ✓
HyperCLOVAXForCausalLMHCXVisionV2ForCausalLM HyperCLOVAX ✓ ✓
IQuestCoderForCausalLM IQuest-Coder ✓
Lfm2MoeForCausalLM LFM 2.5
LlamaForCausalLM Llama 3.1, Llama 3.1-Nemotron etc. ✓
MiMoForCausalLM MiMo-RL ✓
MiniMaxM2ForCausalLM MiniMax-M2 ✓
MistralForCausalLMMistral3ForConditionalGeneration Mistral, Ministral 3, Devstral 2 etc. ✓ ✓
MixtralForCausalLM Mixtral ✓
NanoChatForCausalLM NanoChat
NemotronHForCausalLM NemotronH, Nemotron-3NEW ✓
Olmo3ForCausalLM Olmo 3.1
OlmoHybridForCausalLM Olmo-Hybrid
Phi3ForCausalLM Phi3, Phi4 ✓
Qwen2ForCausalLMQwen2_5_VLForConditionalGeneration Qwen 2, Qwen 2.5, Qwen 2.5 VL ✓ ✓
Qwen3ForCausalLMQwen3MoeForCausalLM Qwen 3 ✓
Qwen3NextForCausalLM Qwen 3-Next
Qwen3VLForConditionalGeneration Qwen 3-VL ✓ ✓
Qwen3VLMoeForConditionalGeneration Qwen 3-VL MoE ✓ ✓
Qwen3_5ForConditionalGeneration Qwen 3.5, Qwen 3.6 ✓ ✓
Qwen3_5MoeForConditionalGeneration Qwen 3.5 MoE, Qwen 3.6 MoE ✓ ✓
SeedOssForCausalLM Seed-OSS ✓
SmolLM3ForCausalLM SmolLM ✓
SolarOpenForCausalLM SolarOpen ✓
Step3p5ForCausalLM Step 3.5 Flash ✓
Step3p7ForConditionalGeneration Step 3.7 Flash ✓ ✓