basal-omni: Run the Decision Model Locally on an RTX 4090

👁 116 reads

basal-omni is Remek Kinas’s Polish decision model: it does not write text, but selects an answer from a list and reports how confident it is. Since October 19, smaller versions have been runnable without a server GPU, using Ollama, llama.cpp or a Mac. I ran the mini version from a GGUF file on my RTX 4090 and tested it on the same set of Polish support tickets I had previously used to evaluate Jev and other decision models. Here are the setup steps that worked for me, the hardware requirements and the results.

Which version should you use on your hardware?

All files are hosted on Remek’s Hugging Face account under the Apache-2.0 license. The GGUF and MLX versions support text only, not images or audio.

The author released only the Q8_0 and Q6_K quantizations because they were the only ones to pass his fidelity check against the original. Across 10,606 decisions with 2-10 options, Q8_0 chose the same answer as the bf16 version in 98.7% of cases; Q6_K did so in 97.1%. Q5_K_M and Q4_K_M did not pass. The largest ultra version (31 billion parameters) has no author-provided GGUF file.

How to set it up on Windows with an NVIDIA GPU and Ollama

This is how I set it up on October 19, 2026, using Windows 11, Ollama 0.40.5 (version 0.35 or later is required) and Python 3.12 through uv.

1. Set up the environment and basal 2.2 engine. In Ollama mode, Ollama runs the model on the GPU, so the CPU version of PyTorch was enough for me:

   uv venv --python 3.12 env
   uv pip install --python env/Scripts/python.exe torch --index-url https://download.pytorch.org/whl/cpu
   uv pip install --python env/Scripts/python.exe "basal @ https://github.com/rkinas/basal/archive/refs/tags/v2.2.0.tar.gz"

2. Download the model (about 8 GB). You need the Q8_0 file and a few small supporting files: Modelfile, tokenizer, calibration and basal.json. I downloaded them individually, but the hf tool, installed with the engine, makes this easier:

   hf download Remek/basal-omni-1.0-mini-GGUF --local-dir basal-omni-1.0-mini-GGUF --exclude "*Q6_K*"

3. Register it in Ollama using the ready-made Modelfile from the repository:

   cd basal-omni-1.0-mini-GGUF
   ollama create basal-omni-1.0-mini:q8_0 -f Modelfile.Q8_0
   cd ..

4. Start the basal server before Ollama:

   basal-serve --model ./basal-omni-1.0-mini-GGUF --mode ollama --ollama-model basal-omni-1.0-mini:q8_0 --port 8000

5. Send your first query (Python, using only the standard library). The format is the same as Jev’s API, with the /v1/systemone endpoint. The example below is a translation of the original Polish test prompt; it was not retested in English:

   import json, urllib.request
   body = {"state": "The dashboard has been down since 9:00 with a 500 error, and the entire sales team is blocked.",
     "questions": {
       "department": {"type": "choice", "instructions": "Which department should handle this ticket?",
         "criteria": {"payments": "payments and invoices", "outage": "something is technically broken",
                      "sales": "pre-purchase questions"}},
       "urgent": {"type": "noul", "instructions": "Does this need a response today?",
         "criteria": {"true": "yes", "false": "no"}}}}
   req = urllib.request.Request("http://127.0.0.1:8000/v1/systemone",
     json.dumps(body, ensure_ascii=False).encode("utf-8"),
     {"Content-Type": "application/json; charset=utf-8"})
   print(json.load(urllib.request.urlopen(req))["answers"])

You get the selected option, the probability of each option and a confidence score (confidence). For a yes-or-no question, you also get the probability of a “yes” answer. Here are two pitfalls I encountered or that the author describes:

• Polish characters in the Windows console. A query sent with curl from Git Bash failed with a UTF-8 decoding error. Sending it from Python with explicit encoding worked without problems.
• Running ollama run on its own is not enough. The model will return free-form text only. basal-serve calculates typed answers and their probabilities by reading the probabilities of the option codes from a single token.
• Limits in Ollama mode: A maximum of 20 options per question and 4,096 tokens per query. For longer documents and up to 255 options, the author recommends llama.cpp mode.

Using llama.cpp or running basal on a Mac

• llama.cpp in-process: Install with the basal[gguf] extra (llama-cpp-python 0.3.36 or later), then run basal-serve --model ./basal-omni-1.0-mini-GGUF --mode gguf --gguf basal-omni-1.0-mini-GGUF/basal-omni-1.0-mini-Q8_0.gguf --port 8000. On an NVIDIA GPU, llama-cpp-python needs to be built with CUDA support. I did not test this setup.
• Mac with Apple silicon: Use --mode mlx with the basal[mlx] extra and the -MLX repository, or run GGUF through Metal. According to the author, mini Q8_0 runs on a Mac with 16 GB of memory. On an M4 Pro, the MLX mini version achieves 0.73 decisions per second with 8 concurrent queries and 173 ms per single query. I did not test this either; I do not have a Mac.

My test: 50 Polish support tickets and 10 tool decisions

I used the set from October 13 that I had used to evaluate models in the Artificial Analysis ranking: 50 e-commerce support tickets (7 departments and urgency) and 10 conversations in which an assistant has to decide whether to answer on its own, call a tool, ask a follow-up question or refuse. The questions and answer key were unchanged; I did not adapt them for basal. As in the earlier test, I scored department classification on 47 tickets because three answer-key entries turned out to be debatable (see the AI decision models guide for details). basal also missed those three, as did all 11 models tested on October 13.

• The smallest basal model lands in the middle of the pack of larger API-based decision models. It classified one more ticket correctly than Jev 1.13 and two fewer than Microsoft-Decision-1 and Decider. It has 4 billion parameters and runs on a consumer GPU.
• Strong urgency results: 47 of 50, surpassed only by Decider and GPT-6 Luna Decisions, which each scored 48. basal did not flag a graphics card under triple load (probability 0.05) or a sparking coffee maker (0.34) as urgent, but incorrectly flagged cryptocurrency spam.
• Department classification errors: a wedding cake marked “in preparation” (sales instead of delivery), a sparking coffee maker (outage instead of complaint), a mailing-list offer (sales instead of spam), a refund for a canceled order and a non-working discount code.
• Check confidence against your own data. At a confidence score of at least 0.9, basal was correct on 35 of 40 tickets, compared with 37 of 39 for Jev. Confident mistakes included the mailing-list offer (0.95) and the non-working discount code (0.97). That is why the new basal-calibrate tool, which tunes thresholds to your examples, matters.
• Speed: median of 137 ms per decision; the first query took 12 seconds while the model loaded onto the GPU. I measured the API models from Poland on October 13, including network time (Jev took 361 ms), so this is not a like-for-like comparison. But with a local model, you do not pay per token and your data stays on your computer.

The test took 20 seconds of GPU time. I ran one pass, with one option order and the calibration file included with the GGUF. I unloaded the model from memory after the test.

When does running basal locally make sense?

• Your data must stay inside your organization - customer tickets, documents and email.
• You have a high volume of decisions that involve choosing from a list: department, urgency, topic, whether to let text through or which tool to call.
• You have a few hundred labeled examples for calibrating thresholds and automatically passing through only high-confidence decisions.

I covered the model’s origins and its ranking results in basal-omni ranks first among open models on JevBench. For other models you can run yourself, see the local AI hub.

// read morebasal-omni: Remek Kinas’s Model Tops JevBench’s Open ModelsBenchmarksCloudflare Clef: Open Decision Model Tested on RTX 4090BenchmarksZ-Image Turbo vs Qwen-Image 2.1: my RTX 4090 testBenchmarks

Kinetyra