Glossary

A model's weights are numbers. Trained at 16 bits each, a 27B model is about 54GB — more than most machines have. Quantization stores each number in fewer bits: 8, 5, 4, sometimes 2. The file shrinks roughly in proportion, and the model gets a little worse at the same time. Nearly every model you run locally is a quantized copy; the question is never whether to quantize but how far.

Rule of thumb 4-bit is the everyday choice. Below 3-bit, things break in ways you will notice.

SEE ALSO: Bits per weight (bpw) · Q4_K_M, Q6_K, Q3_K_S … · Perplexity (PPL) · KLD / KL divergence

The average number of bits used to store one weight — the honest way to compare formats, because names lie. 'Q4_K_M' is not exactly 4 bits per weight; it mixes precisions internally and lands nearer 4.8. Multiply bits-per-weight by parameter count and divide by eight for a rough file size in bytes.

Rule of thumb Comparing two quants? Compare bits per weight, not the labels.

SEE ALSO: Quantization

Full precision — the model as trained, before any compression. FP32 uses 32 bits per weight and is now rare for inference. FP16 and BF16 both use 16 bits but split them differently: BF16 trades decimal precision for a much wider range, which makes training more stable, so most recent models ship in BF16. For running a model the two behave alike. Published benchmark scores are almost always measured here, which is why this site anchors its full-precision band to them.

Rule of thumb Only worth running if you have memory to spare — a 27B model needs about 54GB before context.

COVERS: fp16 · bf16 · fp32

SEE ALSO: Quantization · Bits per weight (bpw)

The standard llama.cpp family, and the names you will meet most often. Read them in three parts: the number is the target bits per weight (Q4 ≈ 4-bit), the K means it uses the smarter 'K-quant' scheme that spends extra bits on the layers that need them, and the final letter is size within that target — S small, M medium, L and XL larger. So Q4_K_M is a medium 4-bit K-quant, and it is the most downloaded quantization in local AI for good reason.

Rule of thumb Q4_K_M is the default. Go to Q6_K if you have room, Q3_K_M only if you must — and prefer Q3_K_M over Q3_K_S, which loses far more than its size saving suggests.

COVERS: Q2_K · Q2_K_S · Q3_K_S · Q3_K_M · Q3_K_L · Q3_K_XL · Q4_K_S · Q4_K_M · Q5_K_S · Q5_K_M · Q6_K · Q6_K_XL

SEE ALSO: Quantization · IQ2_XXS, IQ3_M, IQ4_XS … · GGUF · Bits per weight (bpw) — source

The original, simpler llama.cpp formats — every block of weights gets the same treatment, with no cleverness about which layers matter. K-quants beat them at the same size, so they are mostly historical, with two exceptions: Q8_0 is still the standard 'near-lossless' choice, and Google ships Gemma in Q4_0 because its QAT training targets exactly that format.

Rule of thumb Ignore these unless the model publisher shipped one deliberately, as Google does with Gemma.

COVERS: Q4_0 · Q4_1 · Q5_0 · Q5_1 · Q8_0

SEE ALSO: Q4_K_M, Q6_K, Q3_K_S … · QAT

A second llama.cpp family that squeezes further by first measuring which weights actually matter — running sample text through the model to build an 'importance matrix' — and spending its bits accordingly. At the same file size an IQ quant usually beats the equivalent K-quant, at the cost of being slower to generate on some hardware. The XXS/XS/S/M suffixes run smallest to largest. This site stops at 2-bit; 1-bit IQ builds exist but are not worth your time.

Rule of thumb Worth it when you are squeezing a model into memory it barely fits. At 2-bit expect real damage, whatever the file size suggests.

COVERS: IQ2_XXS · IQ2_XS · IQ2_S · IQ2_M · IQ3_XXS · IQ3_XS · IQ3_S · IQ3_M · IQ4_XS · IQ4_NL

SEE ALSO: Q4_K_M, Q6_K, Q3_K_S … · Quantization

The extreme end: every weight stored as one of three values (-1, 0, +1) for ternary, or effectively one bit. Take an ordinary model down this far with normal quantization and it stops working, so these formats only appear on models purpose-built or purpose-compressed for them. You will see such builds advertised — a 27B in under 4GB, running on a phone — and the quality claims behind them are typically the publisher's own, averaged across their own suite, with no per-dimension breakdown.

Rule of thumb Out of scope here. This site does not rate models at 1-bit, and you should not start there.

SEE ALSO: Quantization · IQ2_XXS, IQ3_M, IQ4_XS … · Bits per weight (bpw) — source

Eight-bit formats used mostly on server GPUs rather than in llama.cpp. FP8 keeps a floating point's exponent so it copes with outlier values; INT8 is a plain integer scale. Both are close enough to full precision that the difference rarely shows up in output quality. Model publishers often release an official FP8 build alongside the original.

Rule of thumb Near-lossless, but half the file is the price. Q6_K gets you most of the way for less.

COVERS: fp8 · int8

SEE ALSO: FP16 / BF16 / FP32 · Quantization

Four-bit floating-point formats, and an easy pair to confuse. MXFP4 is the open Microscaling standard — each small block of weights shares a scaling factor — and it is what OpenAI's gpt-oss models ship in natively. NVFP4 is NVIDIA's variant, with a finer-grained scale that is more accurate but needs recent NVIDIA hardware to run fast. Plain FP4 means neither in particular.

Rule of thumb You do not usually choose these — you get them because the publisher shipped in that format, as gpt-oss does with MXFP4.

COVERS: mxfp4 · nvfp4 · fp4

SEE ALSO: Quantization · FP8 / INT8 — source

Apple's MLX framework has its own quantization, separate from GGUF. The names are refreshingly plain — 4-bit means 4-bit. On Apple Silicon these usually run faster than the GGUF equivalent because MLX is built for the hardware, and mixed-precision variants like OptiQ do better still at the same size.

Rule of thumb On a Mac, try the MLX build of a model before assuming GGUF is your only option.

COVERS: mlx-3bit · mlx-4bit · mlx-8bit

SEE ALSO: MLX · GGUF · Quantization

Two 4-bit schemes from the GPU-server world, used with runtimes like vLLM rather than llama.cpp. Both study the model before compressing it — AWQ protects the weights that activations depend on most, GPTQ compresses layer by layer while correcting the error as it goes. Quality sits in the same neighbourhood as a good 4-bit K-quant.

Rule of thumb Relevant if you run on an NVIDIA GPU through vLLM. On a Mac or in LM Studio, you want GGUF or MLX instead.

COVERS: awq-4bit · gptq-4bit

SEE ALSO: Quantization · Q4_K_M, Q6_K, Q3_K_S …

A measure of how surprised a model is by text it should find ordinary — lower is better. Quantization reports lean on it because it is cheap to compute and sensitive to damage. Its weakness is that it measures prediction, not usefulness: a quant can hold its perplexity and still get noticeably worse at multi-step reasoning.

Rule of thumb Treat a small perplexity rise as 'probably fine' and a large one as 'definitely broken' — but do not read fine detail into it.

SEE ALSO: KLD / KL divergence · Quantization

A direct comparison between the quantized model and the original: for the same input, how far apart are the two probability distributions over the next token? Zero means identical. It is the better measure of quantization damage because it compares against the model you actually wanted, rather than against a text corpus.

Rule of thumb When a quantization writeup reports both, trust the KL divergence over the perplexity.

SEE ALSO: Perplexity (PPL) · Quantization

As a model reads your prompt it computes two vectors per token per layer and keeps them, so it never has to redo that work for tokens it has already seen. That store is the KV cache, and it grows with every token in the conversation. It is the memory cost everyone forgets: the weights are a fixed number you see on the download page, but the KV cache can quietly add tens of gigabytes on a long document. Which is why this site asks what context you need before telling you whether a model fits.

Rule of thumb If a model loads fine and then dies partway through a long chat, this is almost always why.

SEE ALSO: Context window · GQA / MHA · Sliding-window attention (SWA) · MLA · Tokens per second

The maximum number of tokens a model can hold at once — prompt, documents and its own reply together. A token is roughly three quarters of a word. Publishers advertise big numbers (256K is common now) but two things limit what you can actually use: the KV cache for a full window may not fit your memory, and retrieval accuracy usually falls apart long before the window does. Model sheets here show both the advertised window and the measured retrieval score.

Rule of thumb Treat the advertised window as a ceiling, not a promise. Check the long-context score before trusting it.

SEE ALSO: KV cache · RULER · MRCR / needle-in-a-haystack

Classic attention (MHA) gives every attention head its own key and value vectors, which makes the KV cache large. GQA has several query heads share one key/value pair, cutting cache size several-fold for very little quality loss. It is now near-universal — when you see a model with 32 attention heads but only 8 key/value heads, that is GQA, and the 8 is the number that decides your memory bill.

Rule of thumb Fewer key/value heads means a smaller KV cache. It is the single biggest architectural lever on long-context memory.

SEE ALSO: KV cache · MLA

DeepSeek's approach to the same problem: instead of caching keys and values directly, compress them into a much smaller shared latent vector and reconstruct on the fly. The KV cache becomes dramatically smaller — a model using MLA can hold a long context in a fraction of the memory a conventional one needs. GLM-4.7-Flash on this site uses it.

Rule of thumb An MLA model is worth a look if long documents are your workload and memory is tight.

SEE ALSO: KV cache · GQA / MHA

Rather than letting every layer look at the whole conversation, most layers look only at the last N tokens — Gemma uses 1024 — while a few 'global' layers still see everything. The windowed layers stop accumulating cache once they hit their window, so memory growth flattens out. Gemma 4 runs five windowed layers for every global one, which is why it holds long contexts far more cheaply than its layer count suggests.

SEE ALSO: KV cache · Hybrid / linear attention (DeltaNet)

A newer trick: replace most attention layers with a 'linear attention' variant that keeps a fixed-size running state instead of a per-token cache, and leave a few real attention layers to do the heavy lifting. Qwen3.6 uses one full-attention layer in every four, so only 16 of its 64 layers hold a KV cache at all — the reason it needs about 2GB at 32K context where a conventional model its size needs four times that.

SEE ALSO: KV cache · Sliding-window attention (SWA)

The model is split into many small 'expert' sub-networks, and each token is routed to only a few of them. A 30B-A3B model has 30B total parameters but activates about 3B per token, so it generates much faster than a dense 30B. The catch, and it is a big one for local use: all 30B of weights must still be in memory, because the router may pick any expert at any moment. You pay dense memory for sparse speed.

Rule of thumb Choose MoE for speed on a machine with memory to spare — never to save memory.

SEE ALSO: Dense model · Quantization · Memory bandwidth · Tokens per second

The ordinary kind: every parameter takes part in every token. Slower than a mixture-of-experts model of the same total size, but the memory you spend is memory you actually use, and quality per gigabyte is usually better.

SEE ALSO: MoE

Normally a model is trained at full precision and compressed afterwards, and it loses a little in the process. With QAT the publisher fine-tunes the model while simulating the compression, so it learns to tolerate it. A QAT 4-bit build is noticeably better than the same model quantized after the fact. Google ships Gemma this way.

Rule of thumb If an official QAT build exists at the size you want, take it over a community quant.

SEE ALSO: Quantization · Q4_0, Q4_1, Q5_0, Q5_1, Q8_0

On Apple Silicon the CPU and GPU share one pool of RAM, so there is no separate 'graphics memory' to fill and nothing to copy back and forth. That is why a 64GB Mac can run models a 24GB gaming GPU cannot. The catch is the wired limit below. Capacity and speed are separate questions on these machines: the amount is what you choose when buying, but how fast it is read depends entirely on the chip. Across the Apple line that runs from 68GB/s on an M1 to over 1200 on an M5 Ultra, and a 32GB machine alone spans 120 to 614.

SEE ALSO: VRAM · Wired limit (iogpu.wired_limit_mb) · Memory bandwidth

Memory that lives on a discrete graphics card. It is fast and there is not much of it — 8 to 24GB on consumer cards. A model that fits entirely in VRAM runs at full speed; one that does not has to offload.

SEE ALSO: Unified memory · Offloading · Memory bandwidth

When a model is too big for your graphics memory, the runtime keeps the layers that do not fit in ordinary system RAM — and your CPU runs those layers itself. Nothing is shipped back and forth: each processor works on the layers it holds, and the results pass between them. It works, and it is far slower, because system RAM is several times slower than the memory on a graphics card. This site marks any partial-offload fit as tight for that reason. How much slower depends on your system RAM, which is why the picker asks about it.

Rule of thumb A few layers on the CPU is a slowdown. Half the model on the CPU means the CPU is setting the speed, and the card has stopped being why the machine is quick.

SEE ALSO: VRAM · System memory bandwidth · Tokens per second

The bus connecting a graphics card to the rest of the computer. It is often blamed for the cost of offloading, and mostly it should not be: when you split a model deliberately, the layers stay put and the CPU runs its own share, so only small activations cross the bus. PCIe does become the bottleneck in the other case — when a driver quietly spills graphics memory into system RAM behind your back, which is much slower than splitting the model on purpose.

SEE ALSO: Offloading · System memory bandwidth · Tokens per second

macOS will not let the GPU claim all of your unified memory. By default it caps the amount at roughly 75% — about 24GB on a 32GB Mac — and llama.cpp treats that cap as a hard wall, refusing to load anything larger even when you appear to have RAM free. You can raise it with a sysctl command. How far is not settled by anyone: Apple does not document the setting, so every number you will read is somebody's experience rather than guidance. It is a permission, not spare memory — set it past what the system needs and the machine swaps and stutters. One more catch: raising it does not guarantee your runtime will use it. Serving engines apply their own ceiling on top — oMLX, for instance, reserves 6GB for the system by default and takes whichever of that and the sysctl is lower, so on a 32GB Mac anything above 26GB has no effect at all.

Rule of thumb sudo sysctl iogpu.wired_limit_mb=<megabytes>. It reverts on reboot, so treat it as a session: raise it, run the model, put it back.

SEE ALSO: Unified memory — source

Many recent models can work through a problem in a hidden scratchpad before answering. It raises scores on hard maths and code enormously — Qwen3-8B scores 76 on AIME'24 thinking versus 29 not thinking — while costing time and tokens. Because the gap is so large, every provenance note on this site records which mode a number came from.

Rule of thumb Turn it on for maths and debugging, off for chat, and never compare a thinking score against a non-thinking one.

SEE ALSO: AIME

How fast a machine can read its own memory, in gigabytes per second. It is the number that decides generation speed, because producing one token means reading every active weight once and doing only about two arithmetic operations per byte read — so the processor spends its time waiting on memory, not calculating. This is why two machines with the same amount of memory can differ several-fold in speed: a Tesla M40 reads 288GB/s and an RTX 4090 reads 1008GB/s, and both hold a 24GB model.

Rule of thumb Capacity decides what you can run. Bandwidth decides how fast it runs. They are different numbers and vendors advertise the first.

SEE ALSO: Tokens per second · Prompt processing / prefill · VRAM · Unified memory · MoE

How fast text comes out. Roughly ten tok/s reads like a person typing quickly; below about five it is uncomfortable to sit and watch. Estimate it by dividing memory bandwidth by the bytes read per token — the active weights plus the whole KV cache — and taking about half, since real engines never reach peak bandwidth. That KV term is why the same model on the same machine generates noticeably slower at 128K context than at 4K.

Rule of thumb A long context costs speed as well as memory. Both bills come due every token.

SEE ALSO: Memory bandwidth · KV cache · Prompt processing / prefill

Reading what you sent before writing anything back. Unlike generation this part is compute-bound, so it rewards a fast processor rather than fast memory — and the two can point in opposite directions. An NVIDIA DGX Spark ingests a long prompt several times quicker than a Ryzen AI Max+ while generating no faster, and a Mac Studio reverses that. If you paste whole files into a chat you will feel prefill; if you type a sentence and read a long answer you will feel generation.

Rule of thumb Big prompts: watch prompt processing. Long answers: watch tokens per second.

SEE ALSO: Tokens per second · Memory bandwidth · Time to first token

The pause between hitting enter and the first word appearing. It is prompt processing plus whatever the runtime has to set up, so it grows with the length of what you sent, not the length of the reply. A model that generates quickly can still feel sluggish if every turn starts with a long silence — which is what happens when you keep pasting a large document into a machine with weak prefill.

SEE ALSO: Prompt processing / prefill · Tokens per second

How fast your CPU can read ordinary RAM, in gigabytes per second. Set by the memory generation and how many channels the board has — two channels of DDR5 read at about 90GB/s, an eight-channel workstation at about 384. It matters here because any layer your graphics card cannot hold is run by the CPU out of this memory, so it sets the speed of that half of the model. On a machine with no graphics card it sets the speed of all of it.

Rule of thumb More channels beats faster sticks. It is the channel count that moves this number.

SEE ALSO: Offloading · Tokens per second · VRAM

The third memory cost, after the weights and the KV cache, and the one nearly every calculator leaves out. While a model runs it needs somewhere to put its working numbers, and the biggest single piece is a surprise: the final step turns each token into a score for every word in the model's vocabulary, and with a 256,000-word vocabulary that table alone is half a gigabyte. It is set by vocabulary size, model width and batch size — not by how many parameters the model has, so a 400B model needs no more of it than an 8B of the same shape.

Rule of thumb Budget about 1.5-2GB for the runtime before you count weights or cache.

SEE ALSO: KV cache · Flash attention · VRAM

The single-file format llama.cpp and everything built on it reads — LM Studio, Ollama, Jan, KoboldCpp. One file holds the weights, the tokenizer and the chat template together, so there is nothing to assemble. If you are running a model locally on anything other than a Mac-native MLX stack, this is almost certainly what you are downloading.

Rule of thumb The default choice on Windows and Linux, and a fine one on a Mac.

SEE ALSO: llama.cpp · MLX · Q4_K_M, Q6_K, Q3_K_S …

Apple's own machine-learning framework, built for Apple Silicon's unified memory. Models come as a folder of safetensors rather than one GGUF file, and are widely reported as faster than the equivalent GGUF on the same Mac — though by how much is genuinely unsettled: published comparisons range from a clear win on mixture-of-experts models to roughly level, with at least one measurement putting MLX behind llama.cpp. The gap is also said to narrow at long context. mlx-vlm handles the vision models.

Rule of thumb On Apple Silicon, worth trying first — and worth measuring, because the speed claims vary a lot.

SEE ALSO: GGUF · MLX 3-bit / 4-bit / 8-bit · Unified memory — source

The C++ inference engine underneath most local AI tooling. You may never run it directly, but its decisions reach you anyway: the GGUF format, the K-quant and I-quant families, and the bugs. Several known issues on this site are llama.cpp problems rather than model defects — which is why 'update your runtime' is one of the standard fixes here.

Rule of thumb When a model produces garbage right after release, suspect the runtime before the weights.

SEE ALSO: GGUF · Q4_K_M, Q6_K, Q3_K_S … — source

The two most common front ends for running models locally. Ollama is command-line first with a simple pull-and-run model registry; LM Studio is a desktop app with a model browser and a chat window. Both wrap llama.cpp, and LM Studio also runs MLX on Macs. Either is a reasonable starting point.

SEE ALSO: llama.cpp · GGUF

The standard file format for unquantized model weights on Hugging Face, designed so that loading a file cannot execute code — which the older PyTorch pickle format allowed. This is what you get if you download a model's original repository rather than a quantized build.

SEE ALSO: GGUF · config.json

The small file in every model repository that describes its architecture — layer count, attention head counts, context length, which layers use which kind of attention. Unglamorous, but it is where this site's KV-cache figures come from: those numbers are computed from it rather than estimated, and every model sheet links to the file it used.

SEE ALSO: KV cache · GQA / MHA

Instead of relying on what a model memorised during training, you search your own documents first and paste the relevant passages into the prompt. It is the standard fix for two problems at once: models inventing facts, and small models not knowing much. It also sidesteps long-context weakness — retrieving one page beats stuffing a whole book into a window the model cannot really use.

Rule of thumb The highest-value habit for anyone running a small model.

SEE ALSO: Context window · HHEM

An open standard for letting a model call outside services — file access, web search, a calculator, your own scripts — through a consistent interface. It is how a local model gets to do things rather than just talk. Whether it works well depends on the model's tool-calling ability, which this site scores.

SEE ALSO: Tool calling / function calling · BFCL — source

The model emitting a structured request — a function name and arguments as JSON — that your program executes before handing back the result. It is what turns a chat model into an agent, and it is fragile in a way plain chat is not: a malformed call, or an empty one, stops the loop dead. Several known issues on this site are exactly that failure.

SEE ALSO: MCP · BFCL · Chat template

The exact formatting a model expects around your messages — the special tokens marking where the system prompt ends and the user's turn begins. It ships inside the GGUF or alongside the weights. Get it wrong, or let a runtime auto-detect it wrong, and the model gets quietly worse rather than visibly broken: rambling replies, ignored instructions, tool calls that never fire.

Rule of thumb A model that seems dumber than its reputation is often running the wrong template.

SEE ALSO: Tool calling / function calling · llama.cpp

A speed trick: a small fast model drafts several tokens ahead and the real model checks them in one pass, accepting the ones it agrees with. Output is identical to running the big model alone, just quicker. MTP is a variant where the model drafts for itself. It is a pure speed feature — and, being newer code, a common source of bugs.

Rule of thumb If generation looks broken, turn this off before blaming the model.

SEE ALSO: llama.cpp

A way of computing attention that never writes the full score table to memory. Without it a model has to materialise one number per token pair per head, which at 128K context is several gigabytes of scratch on top of everything else; with it, a small mask does the same job. llama.cpp turns it on by default now, and every memory figure on this site assumes it is on. If a model that should fit refuses to load at long context, this is the first thing to check.

Rule of thumb Leave it on. Turning it off costs gigabytes at long context and buys nothing.

SEE ALSO: Compute buffer · KV cache · Context window

A hard high-school maths competition, used as a reasoning test because the answers are exact integers and cannot be bluffed. Scores swing enormously with thinking mode on or off, so always check which was used.

SEE ALSO: Thinking / reasoning mode · MATH / MATH-500

A set of competition maths problems, easier than AIME and long the standard maths benchmark. MATH-500 is a commonly used 500-problem subset.

SEE ALSO: AIME

Physics, chemistry and biology questions written by PhDs to be hard to look up — the point is that searching the web does not help. 'Diamond' is the hardest curated subset. A proxy for genuine scientific reasoning rather than recall.

SEE ALSO: MMLU / MMLU-Pro

Multiple-choice questions across 57 subjects — the long-standing general-knowledge yardstick. Top models saturated the original, so MMLU-Pro raised the difficulty and added answer options. Best read as breadth of knowledge, not reasoning.

SEE ALSO: GPQA Diamond · SimpleQA

Small self-contained programming problems — write one function, pass the unit tests. Easy to run, and now largely saturated and partly leaked into training data. A high HumanEval score does not mean a model can work in your codebase: Gemma 3 27B scores 48.8 here and 4.9% on Aider's real-editing benchmark.

SEE ALSO: SWE-bench Verified / Pro · LiveCodeBench · Aider polyglot

Competitive-programming problems collected continuously from contest sites, so the test set postdates most training data and contamination is harder. Versions (v5, v6) mark different collection windows and are not comparable with each other.

SEE ALSO: HumanEval / EvalPlus / MBPP · SWE-bench Verified / Pro

Real GitHub issues from real Python projects: the model gets the repository and the bug report, and must produce a patch that makes the project's own tests pass. The closest thing to measuring useful engineering. 'Verified' is a human-checked 500-task subset; 'Pro' is a harder, more recent one.

SEE ALSO: Terminal-Bench · LiveCodeBench

Tasks completed by driving a real terminal — install this, fix that, make the command succeed. Measures agentic competence over many steps rather than single-shot code generation.

SEE ALSO: SWE-bench Verified / Pro · OSWorld / AndroidWorld / MCP Atlas

Exercises across several languages performed through the Aider coding tool, which requires the model to edit existing files correctly rather than write from scratch. Brutal on models that look strong on HumanEval — the gap between the two is one of the more useful signals on this site.

SEE ALSO: HumanEval / EvalPlus / MBPP · SWE-bench Verified / Pro — source

Measures whether a model calls tools correctly: does it pick the right function, fill the arguments properly, and know when not to call anything at all. The standard reference for tool-calling ability, and the number to check before building an agent.

SEE ALSO: Tool calling / function calling · Tau-Bench / Tau2 · MCP

Multi-turn conversations where the model must follow a business policy while using tools and talking to a simulated customer. Harder and more realistic than single-call tool tests, because it has to stay coherent across a whole exchange.

SEE ALSO: BFCL · Tool calling / function calling

Agentic benchmarks where the model operates a real computer or phone environment — open the app, click the thing, complete the task. Vendor-reported far more often than independently run, so treat the numbers cautiously.

SEE ALSO: Terminal-Bench · Tool calling / function calling

Tests whether a model obeys precise, checkable instructions — 'reply in exactly three bullet points', 'do not use the letter e'. Verifiable by program, so there is no judgement involved. A good proxy for whether a model will respect your formatting demands.

SEE ALSO: Chat template

Vectara's hallucination leaderboard. The model summarises a document, and a detector checks whether the summary states anything the source does not support. Reported as a rate, so lower is better — the one number on this site where a small figure is the good one.

SEE ALSO: SimpleQA · RAG — source

Short factual questions with one correct answer, chosen to be hard enough that models frequently get them wrong. Measures both what a model knows and whether it admits when it does not.

SEE ALSO: HHEM · MMLU / MMLU-Pro

A long-context test that goes beyond finding a single planted sentence: it adds multi-hop tracing, aggregation and multiple needles at once. Models usually score far worse here than their advertised window suggests, which is the point of running it.

SEE ALSO: MRCR / needle-in-a-haystack · Context window

Long-context retrieval: several similar items are hidden in a long input and the model must recall a specific one. Usually quoted at a length such as 128K. Gemma 4 31B scores 66.4% at 128K — the best on this site, and still a third of the needles missed.

SEE ALSO: RULER · Context window

A judged creative-writing evaluation — another model grades the prose against a rubric. Softer than the rest, and the reason creative-writing scores are sparse here: few of these models have been run through it.

SEE ALSO: LiveBench

A general leaderboard that refreshes its questions monthly from recent sources, so contamination stays low. Covers maths, coding, reasoning and instruction-following in one place.

SEE ALSO: MMLU / MMLU-Pro — source

Qualifiers attached to benchmark numbers, and each one changes the score. pass@k allows k attempts and counts success if any works. n-shot means n worked examples were included in the prompt. CoT means the model was told to reason step by step. A pass@5 score is not comparable to a pass@1 score of the same benchmark.

Rule of thumb Always read the qualifier before comparing two numbers.

SEE ALSO: Thinking / reasoning mode

Model makers sometimes quote benchmarks they built themselves — QwenSWEBench, CoWorkBench, an 'agentic coding suite'. These are not necessarily dishonest, but nobody else has run them, the tasks may suit the model's training, and there is no leaderboard to check against. Treat a headline number from a benchmark you have never heard of as a claim rather than a measurement.

Rule of thumb If you cannot find the benchmark run by anyone but its maker, discount it.

SEE ALSO: SWE-bench Verified / Pro · BFCL