Known-issues sheets for local language models Rev 0.1.0 · reviewed every 7 days
llm-picker

Model datasheet · Gemma 4

Gemma 4 E2B

The smallest Gemma 4, at 3.35GB in Google's QAT build — genuinely multimodal on hardware that cannot run anything else in this catalog. Use it for dictation, captioning and simple chat; its own vendor numbers rule out reasoning and tools.

Vendor
Google
Architecture
Dense · 5.1B
Context
131,072 tokens
License
Apache-2.0
Released
2026-03-31
Vision
Yes

→ GGUF (llama.cpp / LM Studio / Ollama) → MLX (Apple Silicon)

Architecture, from its own config.json

35 layers · 12 sliding window · 3 full attention · 20 sharing an earlier layer's cache

Its last 20 of 35 layers have no keys and values of their own — they read those of an earlier layer, so they add nothing to the cache no matter how long the context gets.

8 attention heads over 1 KV heads (GQA 8:1), head dim 256 — its full-attention layers use a different shape again, 1 KV head of 512, which is the shape that decides the cache at long context · mixture of experts, 2.3B of 5.1B active per token · vocabulary 262,144

Where the memory goes: weights by band above, plus a KV cache of 60MB at 8K · 204MB at 32K · 780MB at 128K, plus the runtime's own working memory — dominated by the output projection over that vocabulary.

Inside one layer, from its own tensors

35 × layer

28 sliding-window layers, 7 full-attention layers — the same block either way. What differs is how far back the mask lets attention look, not what the layer is built from.

input_layernormattentionq_projkvsoftmax(q·kᵀ/√d) · vq_norm · k_norm before the dotproduct8 query heads over 1 key/value headso_projpost_attention_layernormpre_feedforward_layernormfeed-forwardgate_proj · up_proj → down_proj12,288 widepost_feedforward_layernorm

The green line down the right of each half is the residual: it leaves the spine above the norm, skips the operation entirely and rejoins at the +. Where attention is drawn open, q, k and v are scaled to their real output widths — a model that shares key and value heads across many query heads shows one wide box beside two narrow ones.

Around the stack: an embedding of 262,144 × 1,536, a final norm, and an output projection that reuses the embedding (tied weights).

Also in every block and not drawn, because their tensor names do not fix where they belong: layer_scalar, per_layer_input_gate, per_layer_projection, post_per_layer_input_norm. Guessing an order is the one thing this diagram will not do.

§1 Characteristics by quantization band

Q2–Q3 — 2–3 bit

2.5GB weights · e.g. Q3_K_M + KV: 60MB at 8K · 204MB at 32K · 780MB at 128K

— Not yet rated · contributions welcome —

Q4–Q5 — 4–5 bit

3.2GB weights · e.g. Q4_0 , Q4_K_M , mlx-4bit + KV: 60MB at 8K · 204MB at 32K · 780MB at 128K
Math & reasoning 2/10
data checked Aug 2026
  • quant-degradation-community — community consensus @ Q4_0 · aggregated · 2026-08-30

    Editorial derivation: QAT Q4_0 (3.35GB) is the intended build; sub-3B-effective models lose more per bit than large ones.

Coding 2/10
data checked Aug 2026
  • quant-degradation-community — community consensus @ Q4_0 · aggregated · 2026-08-30

    Editorial derivation: carried down one point from the full band.

Tool calling / agentic 1/10
data checked Aug 2026
  • quant-degradation-community — community consensus @ Q4_0 · aggregated · 2026-08-30

    Editorial derivation: do not build a tool-using agent on this.

Q6–Q8 — 6–8 bit

4GB weights · e.g. Q6_K + KV: 60MB at 8K · 204MB at 32K · 780MB at 128K

— Not yet rated · contributions welcome —

FP16 — full precision (fp16/bf16)

10GB weights · e.g. bf16 + KV: 60MB at 8K · 204MB at 32K · 780MB at 128K
Math & reasoning 3/10
data checked Aug 2026
Coding 3/10
data checked Aug 2026
Tool calling / agentic 2/10
data checked Aug 2026
  • vendor-model-card — Tau2 average (with tools) = 24.5 @ bf16 · vendor-reported · 2026-08-30

    The vendor's own number says tool use does not work reliably at this size. Documented as a known issue rather than dressed up.

Long context 1/10
data checked Aug 2026

§2 Known issues & what fixes them

ISSUE-01 severe Q2–Q3 / Q4–Q5 / Q6–Q8 / FP16 · tool-calling

Tau2 24.5 — the vendor's own number says roughly three in four tool-use episodes fail. Treat tool calling as unsupported on this model rather than as something to tune.

No fix
No known fix — choose a different model [strong]

No prompt technique recovers this. If you need tools on small hardware, Gemma 4 E4B (42.2) is the minimum and the 12B (69.0) is the first size that actually works.

Evidence · 1 source · anecdotal
ISSUE-02 severe Q2–Q3 / Q4–Q5 / Q6–Q8 / FP16 · factuality, math

MMLU-Pro 60.0 and GPQA Diamond 43.4 are the lowest in this catalog. At 2.3B effective parameters there is simply not enough capacity to store reliable world knowledge — it will answer general-knowledge questions fluently and wrongly.

Fix
RAG (retrieval) [strong]

Give it the source text and let it summarize or extract. It is a competent reader; it is not a reference work.

Fix
Code interpreter / calculator tool [weak]

Arithmetic belongs in a calculator, not in the weights — though note the model's own tool-calling is unreliable, so drive the tool from your harness rather than letting the model choose it.

Evidence · 1 source · anecdotal
ISSUE-03 severe Q2–Q3 / Q4–Q5 / Q6–Q8 / FP16 · long-context

MRCR v2 at 128K is 19.1% — the advertised window is a buffer size, not a working memory.

Fix
RAG (retrieval) [strong]

Retrieve a page at a time.

Evidence · 1 source · anecdotal