Gemma 4 31B Local Setup: RTX 4090 Benchmarks & VRAM

👁 117 reads

Google’s Gemma 4 31B is currently the most capable dense open-weights model that fits entirely on a single 24 GB GPU like the RTX 4090. We ran it locally via Ollama on our test rig: it generates around 42 tokens per second, scores an impressive 9.5 out of 10 in our multilingual non-English tests, but consumes nearly the entire 24 GB VRAM buffer and punishes slow storage drives. Here are our full benchmark results.

Why Gemma 4 31B fits the single-GPU sweet spot

Our goal was to find a heavy-hitting dense model capable of running completely on a single consumer flagship card (RTX 4090 24 GB, Windows 11). Most other recent candidates quickly fall out of contention:

• Ling 3.1 Flash - features 560 billion parameters with no open weights available yet (as detailed in our look at Ling 3.1 Flash).
• Mistral Small 4 - despite the “Small” label, it packs 119 billion parameters, meaning even aggressive quantization cannot squeeze it into 24 GB.
• GLM 5.3 - lacks a lightweight “Air” tier, while GLM-5.3 Flash starts at a massive 320 billion parameters.
• Hy Image 3.5 - Tencent currently offers API preview access only, with no public weights.
• Qwen 3.8 27B - already evaluated in our 7 local AI models benchmark.

Gemma 4 31B represents the largest dense configuration in the Gemma 4 family that fits onto a standard 24 GB gaming card without offloading layers to system memory.

Gemma 4 31B specs and details

The QAT build underwent quantization-aware training targeting 4-bit precision, resulting in substantially lower perplexity loss compared to traditional post-training 4-bit quantization. On a 24 GB VRAM budget, it is the only practical option, as the Q8 variant exceeds 30 GB.

RTX 4090 performance benchmarks

Pumping out 41 to 43 tokens per second easily exceeds human reading speed. In comparison to our earlier local AI model benchmark, the smaller Gemma 4 12B delivered 104 tok/s using just 8 GB VRAM, while Qwen 3.8 27B took 17 GB at roughly half the 12B model’s speed. Gemma 4 31B is naturally the heaviest of the trio, but remains remarkably responsive for daily interactive tasks.

VRAM headroom warning: do not multitask on your GPU

While running our tests, a background image generation batch was initiated in ComfyUI on the same GPU. VRAM instantly saturated (hitting 23.9 out of 24.0 GB), forcing Windows to page model tensors into shared system RAM. Inference speed crashed from 42 tok/s down to 2.7 tok/s, and then to 0.5 tok/s on the subsequent prompt. Gemma 4 31B leaves virtually zero margin on a 24 GB card: running an image generator alongside it is impossible, and heavy browser tabs can steal enough VRAM to degrade performance.

Multilingual evaluation: 5 non-English test tasks

To stress-test complex grammar, inflections, arithmetic, and cultural reasoning beyond basic English, we tested it in Polish across 5 standardized test prompts scored from 0 to 2 points each.

We noted two minor formatting quirks rather than factual flaws: the model heavily favors Markdown styling and rendered correction arrows using LaTeX formatting ($rightarrow$), which looks unparsed in basic terminal clients. Furthermore, during reasoning steps in ollama run, the internal thought chain takes place in English before synthesizing the final non-English response.

Temat: Przeprosiny za opóźnienie w dostawie zamówienia

Szanowny Panie Grzegorzu,

chciałbym najmocniej przeprosić za 5-dniowe opóźnienie w dostawie Pana krzesła biurowego do Szczebrzeszyna. Bardzo nam przykro z powodu zaistniałej sytuacji.

W ramach rekompensaty przygotowaliśmy dla Pana kod rabatowy o wartości 10% na kolejne zamówienie w naszym sklepie: RABAT10.

Mamy nadzieję, że produkt spełni Pana oczekiwania, a ten gest pozwoli nam zachować Pana zaufanie. W razie dodatkowych pytań pozostaję do dyspozycji.

Z poważaniem,

[Twoje Imię i Nazwisko]
[Nazwa Firmy]

Oto poprawiony tekst:

„Wczoraj poszedłem do sklepu, żeby kupić chleb, ale okazało się, że był zamknięty. W poniedziałek na pewno pójdę jeszcze raz, bo mam nadzieję, że będzie otwarty. Moja koleżanka, która tam pracuje, mówiła, że w ogóle nie zamykają przed 18.”

Lista poprawek:
* poszłem $rightarrow$ poszedłem (błąd fleksyjny)
* sklepu żeby $rightarrow$ sklepu, żeby (brak przecinka przed spójnikiem)
* okazało się że $rightarrow$ okazało się, że (brak przecinka przed spójnikiem)
* napewno $rightarrow$ na pewno (pisownia rozdzielna)
* pójde $rightarrow$ pójdę (brak polskiego znaku)
* nadzieje $rightarrow$ nadzieję (brak polskiego znaku)
* że będzie otwarte $rightarrow$ że będzie otwarty (błąd składniowy - sklep jest rodzaju męskiego)
* koleżanka która $rightarrow$ koleżanka, która (brak przecinka przed zdaniem podrzędnym)
* pracuje mówiła $rightarrow$ pracuje, mówiła (brak przecinka zamykającego zdanie podrzędne)
* wogóle $rightarrow$ w ogóle (pisownia rozdzielna)

Oto wyliczenia dla poszczególnych pozycji:

*   Laptop: netto 4200,00 $rightarrow$ VAT (23%): 966,00
*   Książka: netto 59,90 $rightarrow$ VAT (5%): 3,00
*   Usługa szkolenia: netto 1500,00 $rightarrow$ VAT (23%): 345,00

Suma netto: 5759,90
Suma VAT: 1314,00
Suma brutto: 7073,90

BRUTTO = 7073,90

1) Zielona Góra i Gorzów Wielkopolski.
2) Bolesław Prus.
3) 2499 m n.p.m., Rysy.
4) 1410.
5) 16.

Oto wyjaśnienia tych frazeologizmów napisane prostym językiem:

1. Wyjść jak Zabłocki na mydle
*   Znaczenie: Oznacza ponieść klęskę, sromotnie przegrać lub zostać z niczym, mimo że miało być inaczej (często po tym, jak ktoś wierzył w łatwy zysk).
*   Przykład: *Kupiłem ten tani telefon przez internet, ale okazał się zepsuty i sprzedawca nie chce oddać pieniędzy - wyszedłem jak Zabłocki na mydle.*

2. Rzucać grochem o ścianę
*   Znaczenie: Mówić do kogoś, kto zupełnie nie słucha, nie rozumie lub ignoruje nasze argumenty i prośby.
*   Przykład: *Tłumaczę synowi już godzinę, że musi posprzątać pokój, ale to jak rzucanie grochem o ścianę.*

3. Mieć muchy w nosie
*   Znaczenie: Być w złym humorze bez wyraźnego powodu, być marudnym lub poirytowanym.
*   Przykład: *Nie podchodź teraz do Anny, bo od samego rana ma muchy w nosie i na wszystko się gniewa.*

Licensing: what you can build with Gemma 4

Gemma 4 is released under the permissive Apache 2.0 license. This marks an important transition from Gemma 3, which relied on Google’s custom terms of use containing restrictive use-case clauses. Under Apache 2.0:

• You can freely use the model in commercial applications and customer-facing software.
• You are permitted to fine-tune the architecture and redistribute derivative checkpoints, provided license attribution is maintained.
• There are zero API fees or licensing royalties payable to Google, leaving local electricity and compute as your only operational costs.

Step-by-step setup guide for Ollama

1. Install Ollama from ollama.com (we used version 0.40.1). If your models reside on a secondary drive, configure the OLLAMA_MODELS environment variable (e.g., D:AIollama). For more setup tips, check our Ollama guide.
2. Pull the model file (~18 GB):

   ollama pull gemma4:31b-it-qat

3. Extend the loading timeout if using slower storage. By default, Ollama waits 5 minutes before timing out. Over an external USB drive, loading took 8 to 14 minutes, triggering a “timed out waiting for llama-server to start” error on our first try. In PowerShell (make sure to quit Ollama from the system tray first):

   $env:OLLAMA_LOAD_TIMEOUT = "30m"
   ollama serve

   Pro tip: keep this model on a fast internal NVMe SSD whenever possible.

4. Run and benchmark inference in a separate terminal window:

   ollama run gemma4:31b-it-qat --verbose "Explain in 3 sentences how QAT quantization works."

   The --verbose flag reports the eval rate in tokens per second upon completion. To disable the internal reasoning phase, append --think=false.

5. Confirm hardware allocation:

   ollama ps
   nvidia-smi

   The PROCESSOR column in ollama ps should display “100% GPU”. If you notice split CPU/GPU offloading, reduce your context size or terminate other programs occupying VRAM.

6. Benchmark custom context windows via the API:

   curl http://127.0.0.1:11434/api/chat -d "{"model":"gemma4:31b-it-qat","messages":[{"role":"user","content":"Hello"}],"think":false,"options":{"num_ctx":32768}}"

7. Unload the model when finished to free your VRAM:

   ollama stop gemma4:31b-it-qat

Is Gemma 4 31B right for your setup?

• Yes, if you own a 24 GB GPU (RTX 4090, 3090, or 5090) and need top-tier reasoning and multilingual fluency without transmitting sensitive data to external cloud APIs.
• Yes, for confidential enterprise document review, copyediting, and dense summarization where 42 tok/s delivers prompt responsiveness.
• No, if your hardware has 16 GB VRAM or less. Opt for Gemma 4 12B instead; it runs 2.5 times faster while requiring just 8 GB of VRAM.
• No, if you plan to run concurrent local image or video diffusion pipelines on the same graphics card.

For a broader breakdown of text, image, and video models you can host yourself, explore our comprehensive Local AI overview, and check our hardware reference on how much VRAM you need for local LLMs.

Frequently asked questions

How much VRAM does Gemma 4 31B need?

In its QAT 4-bit configuration, Ollama reserves 19.2 GB of VRAM with an 8k context window and 19.3 GB with 32k context. Total card allocation reached 24.0 GB at peak during our tests, making a dedicated 24 GB GPU an absolute requirement.

Can an RTX 4090 run Gemma 4 31B?

Yes. The entire model fits 100% onto the GPU, delivering 40 to 43 tokens per second. However, no other GPU-intensive tasks can run simultaneously without causing severe performance throttling down to 0.5-2.7 tok/s.

How well does Gemma 4 31B handle non-English languages?

Exceptionally well. In our five-part linguistic test suite, it scored 9.5 out of 10, executing complex inflections, contextual idioms, and localized math without factual errors.

Is Gemma 4 cleared for commercial use?

Yes. Gemma 4 is published under the Apache 2.0 license, which grants commercial deployment, modification, and redistribution rights.

Why does Ollama fail to load Gemma 4 31B?

The most common culprit is a storage timeout: Ollama defaults to a 5-minute initialization window. Set OLLAMA_LOAD_TIMEOUT=30m or relocate weights to an NVMe drive. Insufficient free VRAM will also cause initialization failures.

Should you choose Gemma 4 31B or Gemma 4 12B?

On a 24 GB card, choose the 31B model for maximum comprehension and accuracy. On GPUs with 16 GB or less, or whenever pure generation speed is prioritized, choose Gemma 4 12B (which clocks in at 104 tok/s using just 8 GB VRAM).

// read moreKandinsky 6.0 Video: Local AI Video with Audio on RTX 4090Benchmarks7 Local AI Models on RTX 4090 vs Cloud: Non-English Writing TestBenchmarksLM Studio & MCP: How Much VRAM You Really Need for Local AIGuides

Kinetyra