Model datasheet · Qwen3

Qwen3 8B

Superseded

Superseded by Qwen3.5 9B, which covers the same 8–12GB machines one generation on. Kept as the best-documented small Qwen — the Qwen3 technical report publishes thinking and non-thinking scores for every size — and as the source of this site's one per-quant override.

Vendor: Alibaba
Architecture: Dense · 8.2B
Context: 32,768 tokens
License: Apache-2.0
Released: 2025-04-29
Vision: No

→ GGUF (llama.cpp / LM Studio / Ollama) → MLX (Apple Silicon)

Architecture, from its own config.json

36 layers · 36 full attention

32 attention heads over 8 KV heads (GQA 4:1), head dim 128 · dense feed-forward · vocabulary 151,936

Where the memory goes: weights by band above, plus a KV cache of 1.1GB at 8K · 4.5GB at 32K, plus the runtime's own working memory — dominated by the output projection over that vocabulary.

Inside one layer, from its own tensors

36 × full-attention layer

The green line down the right of each half is the residual: it leaves the spine above the norm, skips the operation entirely and rejoins at the +. Where attention is drawn open, q, k and v are scaled to their real output widths — a model that shares key and value heads across many query heads shows one wide box beside two narrow ones.

Around the stack: an embedding of 151,936 × 4,096, a final norm, and a separate output projection.

On the norm names: despite what it is called, post_attention_layernorm is drawn after the residual add here, which is where this family applies it — it normalises the input to the feed-forward, not the output of attention. Models that carry a separate pre_feedforward_layernorm use it the other way, and are drawn that way.

§1 Characteristics by quantization band

Q2–Q3 — 2–3 bit

4GB weights · e.g. IQ3_XXS , Q3_K_M + KV: 1.1GB at 8K · 4.5GB at 32K · 4.5GB at 128K

Math & reasoning

3/10

data checked Aug 2026

quant-degradation-community — KLD delta + degradation reports @ Q3_K_M · aggregated · 2026-08-30
Editorial derivation: multi-step arithmetic degrades sharply at ≤3-bit; consistent community finding across Qwen-family models.

Coding

4/10

data checked Aug 2026

quant-degradation-community — KLD delta + degradation reports @ Q3_K_M · aggregated · 2026-08-30
Editorial derivation: noticeably more syntax errors and logic slips at 3-bit.

Tool calling / agentic

4/10

data checked Aug 2026

quant-degradation-community — community consensus @ Q3_K_M · aggregated · 2026-08-30
Editorial derivation: malformed JSON arguments become common at ≤3-bit.

Instruction following

6/10

data checked Aug 2026

quant-degradation-community — community consensus @ Q3_K_M · aggregated · 2026-08-30
Editorial derivation: IF holds up better than math/coding at low quants.

Q4–Q5 — 4–5 bit

5GB weights · e.g. Q4_K_M , Q5_K_M , mlx-4bit + KV: 1.1GB at 8K · 4.5GB at 32K · 4.5GB at 128K

Math & reasoning

6/10

data checked Aug 2026

quant-degradation-community — KLD/perplexity delta vs bf16 @ Q4_K_M · aggregated · 2026-08-30
Editorial derivation from the full-band score: community KLD data shows Q4_K_M retains most reasoning ability with a small consistent loss.

Coding

6/10

data checked Aug 2026

quant-degradation-community — KLD/perplexity delta vs bf16 @ Q4_K_M · aggregated · 2026-08-30
Editorial derivation: Q4_K_M coding loss vs bf16 is minor per community testing.

Tool calling / agentic

6/10

data checked Aug 2026

quant-degradation-community — community consensus @ Q4_K_M · aggregated · 2026-08-30
Editorial derivation: tool-call formatting stays reliable at 4-bit for this model per community reports.

Instruction following

7/10

data checked Aug 2026

quant-degradation-community — community consensus @ Q4_K_M · aggregated · 2026-08-30
Editorial derivation: IF is not quant-sensitive; carried from full band.

Q6–Q8 — 6–8 bit

7GB weights · e.g. Q6_K , Q8_0 + KV: 1.1GB at 8K · 4.5GB at 32K · 4.5GB at 128K

— Not yet rated · contributions welcome —

FP16 — full precision (fp16/bf16)

16GB weights · e.g. bf16 + KV: 1.1GB at 8K · 4.5GB at 32K · 4.5GB at 128K

Math & reasoning

7/10

data checked Aug 2026

vendor-model-card — AIME'24 (thinking mode) = 76 @ bf16 · vendor-reported · 2026-08-30
Qwen3 Technical Report Table 17. Non-thinking mode drops to 29.1 — the score reflects both modes.
vendor-model-card — MATH-500 (thinking mode) = 97.4 @ bf16 · vendor-reported · 2026-08-30

Coding

6/10

data checked Aug 2026

vendor-model-card — LiveCodeBench v5 (thinking mode) = 57.5 @ bf16 · vendor-reported · 2026-08-30
Non-thinking mode: 22.8. Model absent from the Aider polyglot leaderboard as of 2026-08-30.

Tool calling / agentic

6/10

data checked Aug 2026

vendor-model-card — BFCL v3 (thinking mode) = 68.1 @ bf16 · vendor-reported · 2026-08-30
Non-thinking mode: 60.2.

Instruction following

7/10

data checked Aug 2026

vendor-model-card — IFEval strict prompt (thinking mode) = 85 @ bf16 · vendor-reported · 2026-08-30

Factuality

6/10

data checked Aug 2026

vectara-hallucination — HHEM hallucination rate % (lower is better) = 4.8 @ bf16 · aggregated · 2026-08-30
95.2% factual consistency, 99.9% answer rate.

Long context

6/10

data checked Aug 2026

vendor-model-card — RULER @128K = 77.4 @ bf16 · vendor-reported · 2026-08-30
RULER @32K: 91.2 — accuracy degrades past 64K.

§2 Per-quant deviations

IQ3_XXS · Math & reasoning drops to 2/10

IQ3_XXS is materially worse than Q3_K_M for arithmetic despite sharing the band — the extreme compression hits attention precision hardest.

§3 Known issues & what fixes them

ISSUE-01 severe Q2–Q3 · math

At 3-bit and below, multi-step arithmetic degrades sharply — expect dropped carries, wrong intermediate steps, and confidently wrong answers.

Fix

Code interpreter / calculator tool [strong]

Route calculations to a Python tool; the model plans fine, it just can't execute arithmetic reliably at this quant.

Fix

Move up a quant band [strong]

Q4_K_M needs only ~7GB and recovers most math ability.

Evidence · 2 sources · community-consensus

llama.cpp quantization quality discussions (KLD/perplexity by quant type) (github, 2026-08-30)
r/LocalLLaMA recurring consensus on ≤3-bit math degradation (reddit, 2026-08-30)

ISSUE-02 moderate Q2–Q3 · coding, tool-calling

At ≤3-bit, code gains subtle logic slips and tool calls start producing malformed JSON arguments.

Fix

Move up a quant band [strong]

The 4–5 bit band fixes most of this for ~1GB more memory.

Fix

Prompt technique [weak]

Constrain output with explicit format examples; helps somewhat with tool-call formatting.

Evidence · 2 sources · community-consensus

llama.cpp quantization quality discussions (github, 2026-08-30)
r/LocalLLaMA reports on tool-calling reliability at low quants (reddit, 2026-08-30)

ISSUE-03 moderate Q2–Q3 / Q4–Q5 / Q6–Q8 / FP16 · creative-writing

Prose is competent but flat, with repetitive phrasing in longer outputs. This is the model's character, not a quant artifact.

No fix

No known fix — choose a different model

No tool meaningfully changes voice. If prose quality is the goal, Gemma 3 12B is the better pick at similar memory.

Evidence · 1 source · anecdotal

r/LocalLLaMA model-recommendation threads for writing (reddit, 2026-08-30)

ISSUE-04 mild Q2–Q3 / Q4–Q5 / Q6–Q8 / FP16 · instruction-following

Qwen3 models fall into endless repetition loops under greedy decoding — the model card itself warns 'DO NOT use greedy decoding'. Quantized builds hit it more often.

Fix

Prompt technique [strong]

Use the card's sampler settings (temp 0.6 / top_p 0.95 / top_k 20 in thinking mode; 0.7 / 0.8 / 20 non-thinking) and presence_penalty up to 1.5 if loops persist.

Evidence · 2 sources · community-consensus

Qwen3 model card (greedy-decoding warning + recommended samplers) (vendor, 2026-08-30)
llama.cpp issue: Qwen3 GGUF repetition reports (github, 2026-08-30)