GGUF Quantization Tested: Q4_K_M vs Q6_K vs Q8_0

👁 113 reads

When downloading local LLMs, GGUF files give you a choice between quantization tags like Q4_K_M, Q6_K, and Q8_0, indicating how aggressively the weights are compressed. In short: Q4_K_M is the practical sweet spot for most systems, Q5_K_M and Q6_K match uncompressed output almost perfectly, and Q8_0 makes sense only when VRAM is not a constraint. To understand where these files fit in your offline stack, check out the local AI models overview.

Quick Summary: What to Choose

• Q4_K_M - The smallest practical quantization where output quality remains intact. For the 11B model, Q4_K_M occupies 6.72 GB and matches the top token of Q8_0 in 94.1% of test evaluations.
• Q5_K_M and Q6_K - If your GPU memory allows it, this is the optimal compromise. The perplexity gap compared to Q8_0 falls within measurement error.
• Q8_0 - Practically identical to the original unquantized weights (96.7% token match to F16 on 1.5B), but nearly double the file size of Q4_K_M and noticeably slower.
• Q3_K_M and below - Only suitable for larger models as an absolute last resort. A 1.5B model compressed to Q2_K failed completely, solving 0 out of 12 benchmark tasks.
• A larger model at Q4 beats a smaller model at Q8. Bielik 11B at Q3_K_M (5.40 GB) solved 11 of 12 tasks, while Bielik 4.5B at Q8_0 (5.06 GB) solved only 5.

What is GGUF?

GGUF is a binary container format created by the ggml-org development team (the authors of llama.cpp) in August 2023. A single file bundles the model weights along with all metadata necessary for execution: tensor architecture, tokenizer parameters, context window limit, and chat template. Because of this unified structure, the exact same file runs identically in llama.cpp, in Ollama vs LM Studio setups, or inside Open WebUI with Ollama.

The specification follows a structured naming standard: ModelName-Size-Tuning-Version-Encoding.gguf. For example, in Bielik-11B-v3.0-Instruct.Q4_K_M.gguf, the final segment describes the quantization format.

Understanding GGUF Quantization: Q4_K_M, Q6_K, Q8_0, F16, and IQ

Model quantization compresses floating-point weights into fewer bits. The base model stores each weight in 16 bits (F16 or BF16). Q8_0 stores weights using roughly 8.5 bits per weight, whereas Q4_K_M drops down to under 5 bits. Smaller files load into less VRAM and generate tokens faster because inference speed is primarily bound by memory bandwidth. The trade-off is a slight precision error across the model layers.

The bits-per-weight numbers come directly from the llama-quantize documentation (measured on Llama 3.1 8B). The K designation indicates k-quants, where weights are grouped into super-blocks of 256 and the block scales are quantized themselves. Suffixes like _S, _M, _L indicate mixed precision: Q4_K_M keeps critical attention and feed-forward tensors at Q6_K precision, whereas Q4_K_S quantizes all weights uniformly at Q4_K. IQ denotes importance matrix quants with non-linear quantization grids, effective below 4 bits. An imatrix (importance matrix) is computed over representative text data to protect the most influential weights from quantization noise.

Benchmarking Quantization on an RTX 4090

On 17 October 2026, I benchmarked 17 GGUF variants on my test machine: an RTX 4090 24 GB, Intel Core i9-14900KF, 64 GB RAM, running Windows 11. I used llama.cpp with CUDA, offloading 100% of the layers to the GPU.

• Model files: Official F16 and Q8_0 files for Bielik 1.5B v3.0, and official Q8_0 files for Bielik 4.5B and 11B sourced from speakleash on Hugging Face (all SHA256 checksums verified). I generated intermediate quantizations using llama-quantize: 1.5B derived from F16, and 4.5B and 11B derived from Q8_0 to remain within download limits. For 11B, I also included the official Q4_K_M package downloaded through Ollama.
• Perplexity and token match: Evaluated using llama-perplexity at context length 2048 against the opening excerpt of Teresa Hennert's Romance by Zofia Nalkowska (roughly 7,900 words, public domain via Wolne Lektury). Token match indicates the percentage of tokens where the model's highest-probability prediction matched the unquantized or Q8_0 reference.
• Generation speed: Measured via llama-bench, evaluating 128 generated tokens across 3 runs.
• VRAM usage: The physical allocation increase reported by nvidia-smi with llama-server running at a 4096 context window.
• Reasoning tasks: A standardized battery of 12 deterministic reasoning tests (grammar, formatting, percentages, JSON output, phrasing) evaluated at temperature 0.

Bielik 1.5B: F16 Down to Q2_K

At 1.5B parameters, a model struggles with factual recall regardless of compression. In F16, it misattributed authorship in literary prompts, while Q3_K_M misidentified historical figures entirely. At Q2_K, the model completely disintegrated, repeating the token "1997" continuously during testing.

Bielik 11B: Q8_0 Down to Q3_K_M

The official Q4_K_M package and my locally quantized Q4_K_M file yielded practically identical results (108.0 vs 109.1 tok/s, both scoring 10/12 on tasks).

Key Findings

• Down to Q5_K_M, perplexity variations are smaller than the measurement error margin (+/-0.10 for the 11B architecture). The first measurable performance decline appears at Q4_K_M in smaller models and at Q3_K_M in 11B models.
• Larger architectures withstand weight quantization significantly better. Q4_K_M maintained 94.1% token fidelity on 11B, compared to 83.6% on 1.5B and 82.7% on 4.5B.
• Importance matrices provide free quality improvements. Adding an imatrix to 1.5B Q4_K_M kept the exact same 0.97 GB file footprint while elevating token match from 83.6% to 88.1%.
• Deterministic prompt suites do not reliably differentiate between 11B quants; score variations between 9 and 11 out of 12 are largely statistical noise on single responses. Token match percentages and perplexity tests measure quantization impact far more accurately.
• Perplexity on Prus's The Doll scored 1.20 on Bielik 11B, proving the model had memorized the training text. That is why I used Nalkowska's text for reliable benchmarking. Do not compare perplexity values between 1.5B and 11B directly, as they use different tokenizers (evaluating at 14,767 and 23,760 tokens respectively).

VRAM Recommendations: 8, 12, 16, and 24 GB

VRAM figures listed below account solely for model memory in llama-server. Running a Windows desktop and a browser generally claims an additional 1 to 2 GB of GPU overhead.

• 8 GB VRAM: Run Bielik 11B at Q3_K_M (6.3 GB allocated at 4096 context) or Bielik 4.5B at Q8_0 (5.3 GB). The 11B model in Q4_K_M takes 7.5 GB, which leaves virtually no safety margin on an 8 GB card.
• 12 GB VRAM: Run Bielik 11B at Q4_K_M or Q5_K_M (7.5 to 8.6 GB), leaving sufficient head-room for expanding context buffers.
• 16 GB VRAM: Run Bielik 11B at Q6_K (9.7 GB) or stick with Q4_K_M to expand the context window to its full 32,768 tokens (13.0 GB total allocation).
• 24 GB VRAM: Run Bielik 11B at Q8_0 with a full 32,768 context buffer (17.7 GB total allocation), or move up to larger models quantized to Q4_K_M.

How to Benchmark Quantization Yourself

1. Download the CUDA-enabled release of llama.cpp from the official ggml-org GitHub repository and extract it to a directory like D:\llama-cuda.
2. Download two or three quantization variants of the same model into one folder using Hugging Face CLI: hf download speakleash/Bielik-11B-v3.0-Instruct-GGUF Bielik-11B-v3.0-Instruct.Q4_K_M.gguf --local-dir D:\modele.
3. Verify your download checksums with Get-FileHash D:\modele\*.gguf against the official repository hashes.
4. Prepare an uncompressed UTF-8 .txt evaluation file containing several thousand words that the model has not memorized.
5. Execute the PowerShell benchmark script below to output inference speeds, file sizes, and perplexity metrics directly to console and CSV.

PowerShell Benchmark Script

I tested this script on 17 October using two Bielik 1.5B files. Running strictly on the CPU (-Warstwy 0), Q4_K_M measured a perplexity score of 54.31, aligning with the 54.27 measurement gathered on GPU during primary testing.

# porownaj-kwantyzacje.ps1 - porównanie kwantyzacji GGUF w llama.cpp (
# Dla każdego pliku .gguf w folderze: rozmiar, szybkość (llama-bench: prompt 512 i generowanie 128 tokenów, 3 powtórzenia)
# i opcjonalnie perplexity na Twoim pliku tekstowym (llama-perplexity, kontekst 2048). Wynik: tabela + plik CSV.
#
# Użycie (PowerShell):
#   powershell -ExecutionPolicy Bypass -File .\porownaj-kwantyzacje.ps1 -Modele D:\modele -Llama D:\llama-cuda -Tekst D:\tekst.txt
# -Modele  folder z plikami .gguf (różne kwantyzacje tego samego modelu)
# -Llama   folder z llama-bench.exe i llama-perplexity.exe (paczka llama.cpp z GitHuba ggml-org, wersja CUDA dla kart NVIDIA)
# -Tekst   (opcjonalnie) plik .txt w UTF-8, najlepiej 5-20 tysięcy słów po polsku; bez niego tylko szybkość
# -Warstwy (opcjonalnie) ile warstw na GPU, domyślnie 99 = wszystkie
param(
    [Parameter(Mandatory = $true)][string]$Modele,
    [Parameter(Mandatory = $true)][string]$Llama,
    [string]$Tekst = "",
    [int]$Warstwy = 99
)
$ErrorActionPreference = "Continue"   # llama.cpp pisze logi na stderr - "Stop" w PowerShell 5.1 przerwałby skrypt
$bench = Join-Path $Llama "llama-bench.exe"
$ppl = Join-Path $Llama "llama-perplexity.exe"
if (-not (Test-Path $bench)) { throw "Nie ma $bench - podaj folder z rozpakowanym llama.cpp w -Llama" }
$pliki = Get-ChildItem -Path $Modele -Filter *.gguf | Sort-Object Length -Descending
if ($pliki.Count -eq 0) { throw "W folderze $Modele nie ma plików .gguf" }
if ($Tekst -and -not (Test-Path $Tekst)) { throw "Nie ma pliku tekstowego $Tekst" }

$wyniki = @()
foreach ($p in $pliki) {
    Write-Host "== $($p.Name)" -ForegroundColor Cyan
    $w = [ordered]@{ Plik = $p.Name; GB = [math]::Round($p.Length / 1e9, 2); Prompt_tok_s = ""; Generowanie_tok_s = ""; Perplexity = "" }

    # 1. Szybkość: llama-bench zwraca JSON (jedna pozycja dla pp512, druga dla tg128)
    $json = & $bench -m $p.FullName -p 512 -n 128 -r 3 -ngl $Warstwy -o json 2>$null | Out-String
    $start = $json.IndexOf("[")
    if ($start -ge 0) {
        foreach ($e in ($json.Substring($start) | ConvertFrom-Json)) {
            if ($e.n_prompt -gt 0) { $w.Prompt_tok_s = [math]::Round($e.avg_ts, 1) } else { $w.Generowanie_tok_s = [math]::Round($e.avg_ts, 1) }
        }
    } else { Write-Host "llama-bench nie zwrócił wyniku (za mało VRAM? spróbuj -Warstwy 20)" -ForegroundColor Yellow }

    # 2. Perplexity (im niżej, tym lepiej; porównuj tylko pliki tego samego modelu na tym samym tekście)
    if ($Tekst) {
        $log = & $ppl -m $p.FullName -f $Tekst -c 2048 -ngl $Warstwy 2>&1 | Out-String
        $m = [regex]::Match($log, "Final estimate: PPL = ([\d.]+)")
        if ($m.Success) { $w.Perplexity = [double]::Parse($m.Groups[1].Value, [Globalization.CultureInfo]::InvariantCulture) }
        else { Write-Host "llama-perplexity: brak wyniku (tekst za krótki? potrzeba min. ok. 4100 tokenów)" -ForegroundColor Yellow }
    }
    $wyniki += [pscustomobject]$w
    Writ

If you run models using Ollama, manual conversion is not required. Executing ollama run SpeakLeash/bielik-11b-v3.0-instruct:Q4_K_M pulls the exact identical weight payload distributed on Hugging Face (which I confirmed by verifying the SHA256 layer hashes in the Ollama registry).

Frequently Asked Questions

Sources

• ggml-org: GGUF File Format Specification
• llama.cpp: llama-quantize Documentation and Bit Rates
• llama.cpp PR 1684: k-quants and _S/_M Mix Implementation
• Hugging Face: speakleash/Bielik-11B-v3.0-Instruct-GGUF
• Hugging Face: speakleash/Bielik-4.5B-v3.0-Instruct-GGUF
• Hugging Face: speakleash/Bielik-1.5B-v3.0-Instruct-GGUF
• Ollama Library: SpeakLeash/bielik-11b-v3.0-instruct
• llama.cpp Releases (ggml-org)
• Author benchmarks recorded 17 October 2026: RTX 4090, llama.cpp CUDA build 11194, 17 GGUF variants.

Checked on 17 October 2026.

// read moreQwen3.6-27B on an RTX 4090: Which Quant FitsGuidesLTX-Video Locally: Open Video Model on Your Own GPU, MeasuredGuidesACE-Step 1.5 Review: Local AI Song Generator with VocalsGuides

Kinetyra