basal-omni: Remek Kinas’s Model Tops JevBench’s Open Models

👁 124 reads

Remek Kinas’s basal-omni-1.0-ultra ranks first among models with open weights on JevBench, a benchmark for models that choose from a list of answers and report how confident they are, rather than writing text. It scores 82.5 on Capability Score. Include closed API services, and only Sqwish’s kettei-fat ranks higher, at 86.3. In that combined comparison, basal beats closed decision models from OpenAI and Microsoft, as well as TypeSafe’s Jev, the model that inspired the category. I checked the JevBench leaderboard, the model cards on Hugging Face and who runs the benchmark. I also ran the smallest version on my computer.

What Remek Kinas announced

On October 17, Remek Kinas (@KinasRemek on X) wrote that basal-omni was “TOP1 in the world among open models” and TOP2 among all models on JevBench. He ended the post with: “So it can be done? It can! Polish AI.” He also thanked the benchmark’s author (@airesearch12) for the independent evaluation. X’s counter showed about 266,000 views on October 19.

basal is Kinas’s project. It has its own inference engine, basal-serve, and two model families: basal-2.0, built on the Polish Bielik v3, and basal-omni, built on Google’s Gemma 4. basal-omni can process text, images and audio (ultra does not support audio), but, like other decision models, it does not generate sentences. It receives a state, such as the contents of a support ticket, along with questions with fixed answer options, then returns the probability of each option in a single pass. TypeSafe pioneered this approach with its Jev model. I wrote about it in September in TypeSafe’s Jev, and cover the category in my guide to AI decision models.

JevBench results

JevBench gives each system the same 1,500 decisions and measures four things: accuracy (Intelligence), calibration - whether stated confidence matches actual accuracy - speed and cost per 1,000 decisions. Its main open-model leaderboard is ranked by Capability Score, the average of accuracy and calibration. Only “Jev-class” systems can enter: they must cost no more than twice as much as Jev and respond no more than twice as slowly.

On the separate API-services leaderboard, which uses the same measure, Sqwish’s kettei-fat leads at 86.3. Other entries include Liquid AI’s d1-ultra (80.5), Jev 1.13.0 (77.1), Jev 1.14.0 (75.0), OpenAI Decisions on GPT-6 Luna (73.5) and Microsoft-Decision-1 (72.2, marked preliminary). With a score of 82.5, basal would rank second if the two leaderboards were combined.

What “TOP1” means - and what it doesn’t

• It means first place for Capability Score, which combines accuracy and calibration. In JevBench’s official composite score, where speed and cost each also count for 25%, basal-omni-1.0-ultra ranks third among 184 open systems. Two much smaller 4B models, H2O-Lightning-4B v1.1 and Torchcast Decision 4B, rank higher. They cost less to run but are notably less accurate (60.0 and 60.3).
• The ranking changes from version to version. When basal appeared on the leaderboard on October 17 (version v1.6.28), the release notes placed it fourth out of 45 in that version’s task sample. The current order is shown on leaderboard v1.7.46, dated October 18.
• JevBench reports ultra’s speed on its own model card: median 0.102 seconds. It costs USD 0.045 per 1,000 decisions, 38% more than Jev, but remains within the Jev-class limit.
• Only ultra appears in the table. The smaller basal-omni models, mini and max, do not have separate JevBench scores.

Who runs JevBench

JevBench is hosted on Benchmark Heaven. According to its website and GitHub repository, it is an open-source hobby project (MIT license) by Florian Standhartinger, who uses @airesearch12 on X - the same person Kinas thanked. The site says it compares models “independently of TypeSafe AI.” The benchmark’s name and scoring are based on Jev: cost and time limits are measured against Jev, and Jev 1.13.0 is shown as a reference point outside the ranking. I found no information on the site about the author’s connections to TypeSafe or Kinas. Some tasks are public; most are sealed so models cannot train on them. The author documents methodology changes in the notes for each version.

JevBench is one of several leaderboards. On the Artificial Analysis Decision Index, as of October 19, Liquid AI’s d1-ultra leads with 65.2 points, ahead of Jev 1.13. basal-omni had not been evaluated there.

Three sizes and the creator’s own benchmark

Verdict v1 uses a hidden set of decisions in Polish and English across 10 categories, prepared by the creator himself, so these are the developer’s results, not an independent measurement. According to the model cards, with the default 1% error threshold, ultra automatically accepts 56.8% of decisions and sends the rest to a human. The weakest categories across all versions involve rules with exact thresholds and dates. The creator recommends handling those in code.

As of October 19, mini and max are also available in GGUF versions for Ollama and llama.cpp, FP8 for vLLM, and MLX for Mac. They launched alongside the basal 2.2 engine and its new tool for calibrating on your own data: basal 2.2 and basal-calibrate.

Who is Remek Kinas?

Remigiusz Kinas is a Principal AI Researcher at Ingenix.ai, a Warsaw-based company building AI models for biology and clinical research. He is one of the co-creators of the Polish Bielik models and a Kaggle Grandmaster. basal is his separate project: the engine code is on his GitHub (rkinas/basal), and the models are on his Remek account on Hugging Face. The 2.2 release also includes fixes from several community contributors.

I tested the smallest version

On October 19, I ran basal-omni-1.0-mini in GGUF Q8_0 (7.9 GB) through Ollama on an RTX 4090. I tested it on my set of 50 Polish support tickets and 10 tool-use decisions, which I had used to test models from the AA ranking on October 13. The test was conducted in Polish. Mini routed 42 of 47 tickets to the right department, got urgency right for 47 of 50 and chose the right tool in 10 of 10 cases. That is one more correctly routed ticket than Jev 1.13 on the same set, with a local median of 137 ms per decision. Step-by-step instructions and all results: run basal locally: Ollama, llama.cpp and Mac.

Who is it for?

• Businesses that classify large volumes of text, such as support tickets, emails and documents, and do not want to send data to an external API. The Apache-2.0 license allows commercial use.
• Teams building AI agents that need a fast, confidence-scored decision about whether to call a tool, ask a follow-up question or refuse.
• Not for writing. basal will not reply to a customer or write a summary; that still requires a regular language model.

// read morebasal-omni: Run the Decision Model Locally on an RTX 4090AI ToolsPerplexity Decider v1.1 27B: Cost, Benchmarks and APIBenchmarksStep 5 Preview: Specs, Pricing, Benchmarks and Polish TestState of AI

Kinetyra