Known-issues sheets for local language models Rev 0.1.0 · reviewed every 7 days
llm-picker

Model datasheet · Gemma 4

Gemma 4 12B

The 'Unified' one: an encoder-free architecture that takes image and audio straight into the embedding space, at a size that leaves room to breathe on a 16GB machine. Punches near the 26B MoE on reasoning while being the only model here that hears.

Vendor
Google
Architecture
Dense · 11.95B
Context
262,144 tokens
License
Apache-2.0
Released
2026-06-03
Vision
Yes

→ GGUF (llama.cpp / LM Studio / Ollama) → MLX (Apple Silicon)

Architecture, from its own config.json

48 layers · 40 sliding window · 8 full attention

16 attention heads over 8 KV heads (GQA 2:1), head dim 256 — its full-attention layers use a different shape again, 1 KV head of 512, which is the shape that decides the cache at long context · dense feed-forward · vocabulary 262,144

Where the memory goes: weights by band above, plus a KV cache of 608MB at 8K · 992MB at 32K · 2.5GB at 128K, plus the runtime's own working memory — dominated by the output projection over that vocabulary.

Inside one layer, from its own tensors

40 × sliding-window layer

input_layernormattentionq_projk_projv_projsoftmax(q·kᵀ/√d) · vq_norm · k_norm before the dotproduct16 query heads over 8 key/valueheadso_projpost_attention_layernormpre_feedforward_layernormfeed-forwardgate_proj · up_proj → down_proj15,360 widepost_feedforward_layernorm

8 × full-attention layer

input_layernormattentionq_projk_projv_projsoftmax(q·kᵀ/√d) · vq_norm · k_norm before the dotproduct16 query heads over 8 key/valueheadso_projpost_attention_layernormpre_feedforward_layernormfeed-forwardgate_proj · up_proj → down_proj15,360 widepost_feedforward_layernorm

The green line down the right of each half is the residual: it leaves the spine above the norm, skips the operation entirely and rejoins at the +. Where attention is drawn open, q, k and v are scaled to their real output widths — a model that shares key and value heads across many query heads shows one wide box beside two narrow ones.

Around the stack: an embedding of 262,144 × 3,840, a final norm, and an output projection that reuses the embedding (tied weights).

Also in every block and not drawn, because their tensor names do not fix where they belong: layer_scalar. Guessing an order is the one thing this diagram will not do.

§1 Characteristics by quantization band

Q2–Q3 — 2–3 bit

6GB weights · e.g. Q3_K_M + KV: 608MB at 8K · 992MB at 32K · 2.5GB at 128K

— Not yet rated · contributions welcome —

Q4–Q5 — 4–5 bit

7GB weights · e.g. Q4_0 , Q4_K_M , mlx-4bit + KV: 608MB at 8K · 992MB at 32K · 2.5GB at 128K
Math & reasoning 6/10
data checked Aug 2026
  • quant-degradation-community — community consensus @ Q4_0 · aggregated · 2026-08-30

    Editorial derivation: QAT Q4_0 is the intended local build; one point for quant loss, consistent with the family.

Coding 6/10
data checked Aug 2026
Tool calling / agentic 6/10
data checked Aug 2026
  • quant-degradation-community — community consensus @ Q4_0 · aggregated · 2026-08-30

    Editorial derivation: carried from the full band.

Q6–Q8 — 6–8 bit

10GB weights · e.g. Q6_K + KV: 608MB at 8K · 992MB at 32K · 2.5GB at 128K

— Not yet rated · contributions welcome —

FP16 — full precision (fp16/bf16)

24GB weights · e.g. bf16 + KV: 608MB at 8K · 992MB at 32K · 2.5GB at 128K
Math & reasoning 7/10
data checked Aug 2026
  • vendor-model-card — AIME 2026 (no tools) = 77.5 @ bf16 · vendor-reported · 2026-08-30

    GPQA Diamond: 78.8, MMLU-Pro: 77.2. A 12B scoring 77.5 on AIME 2026 is the headline of this release.

Coding 7/10
data checked Aug 2026
Tool calling / agentic 7/10
data checked Oct 2026
  • vendor-model-card — Tau2 average (with tools) = 69 @ bf16 · vendor-reported · 2026-08-30

    Essentially level with the 26B-A4B (68.2) at half the footprint.

  • vendor-model-card — BFCL-v4 (FC) = 59.68 · aggregated · 2026-10-04

    Below Ling-3.0-tiny (62.72) and Qwen3.5-9B (65.83) in the same table, so the outside number is less flattering than Google's Tau2 69.0. Held at 7 because the two benchmarks measure different things and this one is competitor-run; treat 7 as the top of the plausible range. Third-party measurement by inclusionAI, published as an image in the Ling-3.0-tiny model card (column 'Gemma-4-12B-it (Thinking)'). A competitor-run comparison, not Google's number and not a neutral leaderboard; thinking mode, quant not stated (presumed full precision). The -AA rows in the same table are deliberately not recorded.

Long context 4/10
data checked Aug 2026
Creative writing 6/10
data checked Oct 2026
  • eqbench-creative — EQ-Bench Creative Writing v3 rubric score (0–20) = 15.71 · aggregated · 2026-10-04

    Independent; row google/gemma-4-12B-it. Rubric rank 79 of 140 (board median 15.96); by Elo (1288.9, relative, judged pairwise by Claude Sonnet 4.6) it ranks 95. The rubric is the stabler figure and is what is stored. Quant and mode not stated (presumed full-precision API). Data file behind https://eqbench.com/creative_writing.html.

  • eqbench-creative — EQ-Bench Creative Writing Longform score (0–100) = 48.7 · aggregated · 2026-10-04

    Rank 95 of 140 on the longform board (median 59.1). Mid-table prose: solid rubric scores, weaker in pairwise comparison and at chapter length.

Instruction following 7/10
data checked Oct 2026
  • vendor-model-card — IFBench = 73.5 · aggregated · 2026-10-04

    Multi-IF 88.44, LIFEBench 65.30 in the same table. First instruction-following evidence for this model. Third-party measurement by inclusionAI, published as an image in the Ling-3.0-tiny model card (column 'Gemma-4-12B-it (Thinking)'). A competitor-run comparison, not Google's number and not a neutral leaderboard; thinking mode, quant not stated (presumed full precision). The -AA rows in the same table are deliberately not recorded.

§2 Known issues & what fixes them

ISSUE-01 moderate Q2–Q3 / Q4–Q5 / Q6–Q8 · instruction-following

The 12B ships a Multi-Token-Prediction drafter for speculative decoding, and that path has its own bug reports in llama.cpp — enabling it (`-fit on`) can break generation on this size specifically. The speed feature is the risk, not the model.

Fix
Update your runtime [strong]

Run without MTP/speculative decoding until the issue closes, or track a llama.cpp build that has the Gemma4 MTP fix. Plain Q4_0 decoding is unaffected.

Evidence · 2 sources · community-consensus
ISSUE-02 severe Q2–Q3 / Q4–Q5 / Q6–Q8 · instruction-following, coding

Shares the family's rocky local-runtime history: Google's refreshed official GGUFs (2026-07-15) failed to load in LM Studio 0.4.19, and the family produces gibberish past ~230 tokens when KV offload (-nkvo) is on.

Fix
Update your runtime [strong]

Take a re-cut community quant and a current llama.cpp; don't run Gemma 4 with -nkvo.

Evidence · 2 sources · community-consensus
ISSUE-03 moderate Q2–Q3 / Q4–Q5 / Q6–Q8 / FP16 · long-context

The 256K window oversells it: retrieval accuracy at 128K is 43.4% (vendor's own MRCR v2 number), essentially the same as the 26B-A4B. Don't stuff the window and expect recall.

Fix
RAG (retrieval) [strong]

Retrieve and paste the relevant chunk rather than the whole corpus.

Evidence · 1 source · anecdotal