Strata: 125B Model on 12 GB GPU at 94 Tokens/Second

👁 123 reads

The free Strata engine runs the Qwen3.8-Flash-Next model with 125 billion parameters on a standard graphics card with 12 GB of memory and, according to its author, generates about 94 tokens per second on an RTX 5070. The GitHub project has collected over 10,000 stars, but all the numbers so far come from the creator himself.

Strata is an MIT-licensed engine for running models, published by Niko Veit (on GitHub Niko1221/Strata). In early October, the project was covered by AlphaSignal and AI Weekly, among others, and test videos appeared on YouTube. Installation is a single script on Windows or Linux, and the engine exposes a local API compatible with OpenAI and Anthropic.

How 125 Billion Parameters Fit in 12 GB

The trick lies in the model architecture. Qwen3.8-Flash-Next is a mixture of experts model: it has 24,576 “experts,” but uses about 10 of them per token, meaning about 6 billion active parameters. Strata:

• keeps on the GPU the experts the model uses most frequently;
• loads the rest from RAM, and if needed from SSD;
• uses a built-in draft model (speculative decoding), which according to the project speeds up generation by 1.6-1.8x;
• has CUDA kernels rewritten specifically for this one model - this is not a universal replacement for llama.cpp or Ollama.

Requirements and Claimed Performance

First launch heavily loads the computer for 1-3 minutes, as 35-55 GB of data loads into RAM. A long initial prompt takes about a minute to load for every 30,000 tokens.

What to Watch Out For

• No independent testing. All numbers in the README are the author’s claims, and the project itself marks results for RTX 3090 and 40-series as estimates.
• Wide variation in user results. On Reddit there are reports of over 100 tok/s on two RTX 3090s, but also a comment about an average of 20 tok/s on RTX 4070 with 64 GB RAM.
• Low quantization. The fastest results are for the Q2_0 variant, meaning a heavily compressed model. Response quality drops with such compression, and the project provides no quality comparison.
• Rapid changes. The engine went from version 0.1.24 to 0.1.27 in two days, and forks with different requirements are also circulating online.

What This Means for You

If you have a gaming PC with a 12 GB card and 64 GB RAM, you can try running a model of this class without the cloud and without paying for tokens. However, install it like any fresh GitHub project: with the awareness that it’s an early version, ideally on a separate account. If you’re starting your journey with local models, a simpler and more proven path is Ollama - we’ve covered it along with running Llama and the required VRAM. We discuss the Qwen model family, from which Flash-Next comes, in our Qwen guide.

Sources: Niko1221/Strata repository (README), AlphaSignal, AI Weekly, user posts on Reddit and YouTube. Verified October 4, 2026.

// read moreLocal AI vs API: Real Cost of Running Models on Your GPUGuidesRelational Model Training: 305 Seconds on One GPUBenchmarksHow Ai2 Fixed GPU Scheduling Across Thousands of H100 and B200sState of AI

Kinetyra