Known-issues sheets for local language models Rev 0.1.0 · reviewed every 7 days

Model datasheet · Qwen3.8

Qwen3.8 27B

The current Qwen open flagship under 35B: hybrid DeltaNet attention, vision, 256K context, and vendor benchmarks that look absurd — none independently replicated yet. Two weeks old; treat every number as a claim.

Vendor
Alibaba
Architecture
Dense · 27B
Context
262,144 tokens
License
Apache-2.0
Released
2026-08-14
Vision
Yes

→ GGUF (llama.cpp / LM Studio / Ollama) → MLX (Apple Silicon)

§1 Characteristics by quantization band

Q2–Q3 — 2–3 bit

10.7GB weights · e.g. Q3_K_M + KV: 512MB at 8K · 2GB at 32K · 8GB at 128K
Math & reasoning 7/10
data checked Aug 2026
  • quant-degradation-communityGPQA Diamond = 88 @ Q2_K · aggregated · 2026-08-31

    Roughly 88% at 2-bit against roughly 90% at full precision — a real but small loss, and far better than 2-bit usually manages. One point down.

Coding 7/10
data checked Aug 2026
  • quant-degradation-communityTerminal-Bench 2.1 @ Q2_K · aggregated · 2026-08-31

    The study reports a noticeable fall at 2-bit on agentic coding while remaining competitive. Two points down from the measured 4-bit result.

Instruction following 7/10
data checked Aug 2026
Tool calling / agentic 6/10
data checked Aug 2026
  • quant-degradation-communityTerminal-Bench 2.1 @ Q2_K · aggregated · 2026-08-31

    Editorial derivation: two points down. Structured tool output is the first thing to go at 2-bit, and the study's agentic scores fall there.

Q4–Q5 — 4–5 bit

17GB weights · e.g. Q4_K_M , mlx-4bit + KV: 512MB at 8K · 2GB at 32K · 8GB at 128K
Coding 9/10
data checked Aug 2026
Math & reasoning 8/10
data checked Aug 2026
  • quant-degradation-communityGPQA Diamond = 90 @ Q4_K_M · aggregated · 2026-08-31

    Measured at roughly 90% against the full model's roughly 90%. No discount applied because none was observed.

Instruction following 8/10
data checked Aug 2026
Tool calling / agentic 8/10
data checked Aug 2026

Q6–Q8 — 6–8 bit

29GB weights · e.g. Q6_K , Q8_0 + KV: 512MB at 8K · 2GB at 32K · 8GB at 128K
Coding 9/10
data checked Aug 2026
  • quant-degradation-communityTerminal-Bench 2.1 vs bf16 @ Q8_0 · aggregated · 2026-08-31

    Measured, not derived: an independent study found Q4_K_M matching the full model on Terminal-Bench 2.1, so 8-bit sits between two points that are equal. Carried unchanged.

Math & reasoning 8/10
data checked Aug 2026
Instruction following 8/10
data checked Aug 2026
Tool calling / agentic 8/10
data checked Aug 2026

FP16 — full precision (fp16/bf16)

55GB weights · e.g. bf16 + KV: 512MB at 8K · 2GB at 32K · 8GB at 128K
Coding 9/10
data checked Aug 2026
  • vendor-model-cardLiveCodeBench v6 = 90.3 @ bf16 · vendor-reported · 2026-08-31

    Terminal-Bench 2.1: 73.0, SWE-bench Pro: 61.7, NL2Repo-Bench: 42.3. The highest figures in this catalogue on all three — Qwen3.6 27B manages 83.9 on LiveCodeBench and Ornith-1.5 35B-A3B 68.5 on Terminal-Bench.

  • community-consensus — Code Arena WebDev leaderboard placement @ bf16 · aggregated · 2026-08-31

    Independent, and the reason this cell moved from 8 to 9: #9 overall on Code Arena's WebDev board and the only model in its size class in the top 10, plus the #1 open-weight slot on Arena.ai's Image-to-WebDev board. Third-party evaluation of the coding claim now exists; it did not on 2026-08-30.

  • artificial-analysis — Agentic Index placement (link-only — no numbers republished) @ bf16 · aggregated · 2026-08-31

    Independent aggregator, cited by link under their terms. Places it immediately behind a 2.8T-parameter model on agentic work.

Math & reasoning 8/10
data checked Aug 2026
  • vendor-model-cardGPQA Diamond = 89.2 @ bf16 · vendor-reported · 2026-08-31

    Humanity's Last Exam: 30.8. The GPQA figure was independently reproduced at ~90% in a third-party quantization study, which is the first outside confirmation of any Qwen3.8 number.

Instruction following 8/10
data checked Aug 2026
  • vendor-model-cardIFBench = 79.5 @ bf16 · vendor-reported · 2026-08-31

    Above Nemotron 3 Nano's 71.5 on the same benchmark. Also reproduced independently in the quantization study below. IFBench is program-checkable, so there is little interpretive room.

Tool calling / agentic 8/10
data checked Aug 2026
  • vendor-model-cardOSWorld-Verified = 84.3 @ bf16 · vendor-reported · 2026-08-31

    AndroidWorld 81.9, WebArena-Verified 64.8 — none of them Qwen-built, unlike the QwenSWEBench and CoWorkBench numbers on the same card, which are excluded here for that reason.

§2 Known issues & what fixes them

ISSUE-01 moderate Q2–Q3 / Q4–Q5 / Q6–Q8 / FP16 · instruction-following

Ships with reasoning effort defaulted to 'xhigh' and wildly overthinks simple prompts — one reviewer measured 21 minutes and 22K reasoning tokens on a trivial question at default settings.

Fix
Prompt technique [strong]

Turn the reasoning-effort setting down for everyday use; save xhigh for problems that earn it.

Evidence · 1 source · anecdotal
ISSUE-02 severe Q2–Q3 / Q4–Q5 / Q6–Q8 · coding, instruction-following

Its new DeltaNet architecture had a CUDA bug producing silent garbage output on older llama.cpp builds — quantized local runs on stale runtimes can look broken when the model is fine.

Fix
Update your runtime [strong]

Use llama.cpp ≥ b10419 (or a runner built on it). This is a tooling bug, not a model defect.

Evidence · 2 sources · community-consensus
ISSUE-03 mild Q2–Q3 / Q4–Q5 / Q6–Q8 / FP16 · coding, math

Partly resolved. Within two weeks of release the coding claim picked up independent third-party corroboration — a top-10 overall placing on Code Arena's WebDev board and the #1 open-weight slot on Arena.ai's Image-to-WebDev board — and an outside quantization study reproduced the GPQA, IFBench and Terminal-Bench figures across the quant ladder. What remains unmeasured by anyone is factuality and long context: no hallucination benchmark and no RULER or MRCR result exists for this model, so those cells are blank rather than estimated. Note also that the flashiest numbers on the vendor card — QwenSWEBench, CoWorkBench — are Qwen's own benchmarks and are deliberately not scored here.

No fix
No known fix — choose a different model

Coding and reasoning now have outside support. Treat the factuality and long-context blanks as genuinely unknown, not as implied strength.

Evidence · 2 sources · community-consensus