Model datasheet · Nemotron 3.5
Nemotron 3.5 Lightning 30B-A3B
Replaces Nemotron 3 Nano, which it beats on every row of NVIDIA's own comparison — SWE-bench Verified 51.6 against 34.1, Terminal-Bench 24.6 against 8.3. Same hybrid Mamba-MoE shape and the same cheap cache: 6 of 52 layers hold KV, so 256K of context costs about 1.5GB. Licensed OpenMDW rather than the old Nemotron licence.
- Vendor
- NVIDIA
- Architecture
- MoE · 31.6B total / ~3B active
- Context
- 262,144 tokens
- License
- OpenMDW-1.1
- Released
- 2026-08-11
- Vision
- No
→ GGUF (llama.cpp / LM Studio / Ollama) → MLX (Apple Silicon)
Architecture, from its own config.json
52 layers · 46 linear / recurrent · 6 full attention
Only 6 of 52 layers hold a cache at all, which is why its memory grows so much more slowly with context than its size suggests.
32 attention heads over 2 KV heads (GQA 16:1), head dim 128 · mixture of experts, 3B of 31.6B active per token across 6 experts · vocabulary 131,072
Where the memory goes: weights by band above, plus a KV cache of 48MB at 8K · 192MB at 32K · 768MB at 128K, plus the runtime's own working memory — dominated by the output projection over that vocabulary.
Inside one layer
Not drawn. This model names its per-layer norm norm, which fixes no position — it could be the one before attention or the one before the feed-forward, and the name does not say. The memory figures above come from its tensors either way; the block order does not, so it is left out rather than guessed.
§1 Characteristics by quantization band
data checked Sep 2026
- quant-degradation-community — editorial band derivation @ Q3_K_M · aggregated · 2026-09-12
Editorial derivation: two points down from bf16. 3-bit on a 3B-active mixture is the most aggressive combination on this site.
data checked Sep 2026
- quant-degradation-community — editorial band derivation @ Q3_K_M · aggregated · 2026-09-12
Editorial derivation: two points down, and the saving over Q4_K_M is only 5.7GB — rarely the trade worth making.
Q4–Q5 — 4–5 bit
25.5GB weights · e.g. Q4_K_M , mlx-4bit + KV: 48MB at 8K · 192MB at 32K · 768MB at 128Kdata checked Sep 2026
- quant-degradation-community — editorial band derivation @ Q4_K_M · aggregated · 2026-09-12
Editorial derivation: one point down. At 25.5GB this is the band a 32GB machine actually runs.
data checked Sep 2026
- quant-degradation-community — editorial band derivation @ Q4_K_M · aggregated · 2026-09-12
Editorial derivation: one point down; math is the first thing to go under 4-bit on a 3B-active model.
data checked Sep 2026
- quant-degradation-community — editorial band derivation @ Q4_K_M · aggregated · 2026-09-12
Editorial derivation: one point down. Structured calls need token-level precision that a sparse model has less of to spare.
data checked Sep 2026
- quant-degradation-community — editorial band derivation @ Q4_K_M · aggregated · 2026-09-12
Editorial derivation: carried; the cache is fp16 regardless of the weight quant.
data checked Oct 2026
- eqbench-creative — EQ-Bench Creative Writing v3 rubric score (0-20) = 14.39 @ nvfp4 · aggregated · 2026-10-04
Measured, not derived: EQ-Bench ran NVIDIA's NVFP4 build (row nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4), a 4-bit format, so it anchors the Q4-Q5 band. No full-precision row exists, so the full band is left unrated.
Independent; rubric judged by Claude Sonnet 4. Normalised Elo on the same row is 1280.3 (96th of 140, retrieved 2026-10-04; Elo moves as models are added, so the rubric score is the one stored). The longest outputs on the board (11,382 characters on average). Between Gemma 3 12B (14.63) and Gemma 3 4B (13.91) on rubric. Not on the Longform board. Data file behind https://eqbench.com/creative_writing.html.
data checked Sep 2026
- quant-degradation-community — editorial band derivation @ Q6_K · aggregated · 2026-09-12
Editorial derivation: carried from the full band; 6-bit is indistinguishable from bf16 on every published KLD table.
data checked Sep 2026
- quant-degradation-community — editorial band derivation @ Q6_K · aggregated · 2026-09-12
Editorial derivation: carried from the full band.
data checked Sep 2026
- quant-degradation-community — editorial band derivation @ Q6_K · aggregated · 2026-09-12
Editorial derivation: carried from the full band.
data checked Sep 2026
- quant-degradation-community — editorial band derivation @ Q6_K · aggregated · 2026-09-12
Editorial derivation: carried from the full band; long context is not a quant-sensitive dimension here.
FP16 — full precision (fp16/bf16)
65.9GB weights · e.g. bf16 + KV: 48MB at 8K · 192MB at 32K · 768MB at 128Kdata checked Sep 2026
- vendor-model-card — SWE-bench Verified = 51.56 @ bf16 · vendor-reported · 2026-09-12
NVIDIA's own table. Half again the Nemotron 3 Nano it replaces (34.08) and ahead of GPT-OSS 20B (52.44) only narrowly; Qwen3.6 35B-A3B posts 70.12 on the same row.
- vendor-model-card — Terminal-Bench 2.1 = 24.58 @ bf16 · vendor-reported · 2026-09-12
Agentic terminal work, three times the 3 Nano's 8.29 — the largest single gain in the generation, and still well behind Qwen3.6 35B-A3B at 44.38.
data checked Sep 2026
- vendor-model-card — GPQA Diamond (no tools) = 75.44 @ bf16 · vendor-reported · 2026-09-12
Graduate-level science questions. Above the 3 Nano (74.05), below Nemotron 3 Super (78.60) and Qwen3.6 35B-A3B (83.40).
- vendor-model-card — MMLU Pro = 81.94 @ bf16 · vendor-reported · 2026-09-12
Broad multi-task reasoning, vendor-reported.
data checked Sep 2026
- vendor-model-card — PinchBench = 85.37 @ bf16 · vendor-reported · 2026-09-12
Tool-use benchmark on NVIDIA's table, well ahead of the 3 Nano's 66.11.
- vendor-model-card — τ³-bench (Banking) = 9.28 @ bf16 · vendor-reported · 2026-09-12
The other side of the same story: multi-turn tool work in a banking scenario is still near the floor for every model in this size class, Gemma 4 26B-A4B's 14.02 included. The score is held at 5 because the two measurements disagree about how usable this is.
data checked Sep 2026
- vendor-model-card — IFBench (loose) = 71.88 @ bf16 · vendor-reported · 2026-09-12
Ahead of Qwen3.6 35B-A3B (63.71) and level with Nemotron 3 Super (71.92).
data checked Sep 2026
- vendor-model-card — MMLU Pro = 81.94 @ bf16 · vendor-reported · 2026-09-12
Knowledge proxy only — no hallucination benchmark lists this model. 3B active parameters is the real ceiling on what it can recall unaided.
data checked Sep 2026
- vendor-model-card — AA-LCR (value at source) @ bf16 · vendor-reported · 2026-09-12
NVIDIA reports an Artificial Analysis long-context reasoning score; neither the value nor how it compares with other models is republished here. Note this is a different benchmark from the RULER-at-1M figure that made the 3 Nano the long-context pick here, so the two are not comparable and the score is set conservatively.
§2 Known issues & what fixes them
GGUFs built in the first fortnight after release can be wrong. llama.cpp shipped two conversion fixes for this model in late August — one where zero KV-head metadata made the converter read attention blocks as Mamba blocks, one correcting the layer list outright. A file converted before those landed loads and generates, badly.
Update llama.cpp past 2026-08-26 and re-download the GGUF rather than reusing an early one. The ggml-org and bartowski repos were rebuilt after the fixes.
Evidence · 2 sources · community-consensus
- Nemotron 3.5 Lightning Q4_0: zero KV head metadata misclassifies Attention blocks as SSM (github, 2026-09-12)
- conversion : fix Nemotron 3.5 Lightning layers (github, 2026-09-12)
The speculative-decoding draft head is shipped as a separate GGUF, and loading it while offloading experts to the CPU crashes below a certain --n-cpu-moe threshold. Still open as of 2026-10-04; a fix pull request is open but not merged. A second report has attaching an external MTP draft trip an assertion in the Mamba scan during context reservation. This only costs you the speed-up; the model itself runs.
Run without the MTP draft model if you are splitting experts onto the CPU, or keep the whole model resident.
Evidence · 3 sources · anecdotal
- Eval bug: --n-cpu-moe below a threshold crashes MTP draft model load (open) (github, 2026-10-04)
- llama : fix out-of-bounds layer split when all devices report zero free memory (fix for #27717, open) (github, 2026-10-04)
- nemotron_h_moe aborts in ggml_ssm_scan during context reservation (Lightning comment: only with an external MTP draft attached) (github, 2026-10-04)
Every coding, math, tool-calling, instruction-following, factuality and long-context score here is NVIDIA's own, from a table NVIDIA also chose the comparison set for. The only independent leaderboard that lists the model as of 2026-10-04 is EQ-Bench Creative Writing, and it tested the NVFP4 build.
Not a defect — a caveat. Revisit when independent numbers land.
Evidence · 2 sources · anecdotal
- NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 model card (vendor benchmark table) (vendor, 2026-09-12)
- EQ-Bench Creative Writing v3 (lists the NVFP4 build only) (leaderboard, 2026-10-04)
Only 3B of 31.6B parameters are active per token, so there is little redundancy left for 3-bit to eat into. The band saves 5.7GB over Q4_K_M and costs about two points on every quant-sensitive dimension.
Q4_K_M is 25.5GB and fits any 32GB machine with room for a working context. That is the band to run.
Where arithmetic is what broke, hand it to a tool instead of the weights.
Evidence · 2 sources · community-consensus
- llama.cpp quantization quality discussions (KLD tables for sparse MoE) (github, 2026-09-12)
- bartowski GGUF quant sizes for this model (other, 2026-09-12)