LLM Comparison in Polish: 4 Open Models, 12 Tasks, Real Results

👁 111 reads

The open-weight releases available as of October 2026 gave me a useful LLM comparison: four models, the same short prompts, and a language outside the usual English benchmark discussion. My test in Polish took place on October 19 via the API. DeepSeek V4.1 Flash 1017 and Nemotron 3.5 Super VL scored 12/12, Muse Spark Open 11/12, and Step 5 Preview 9/12.

Method: an LLM comparison with checkable tasks

I submitted 48 requests in total, with no repeated trials for an individual task. The same prompts and generation settings went to all four models. A script evaluated the returned text against answer checks. The saved records contain elapsed request time, completion tokens, cost, response text, and the final pass or fail result. What each model is, how big it is and under which license it ships is summarized in my open-weight AI models overview.

The tasks were deliberately short. Some had a unique numerical answer, while the grammar and formatting prompts allowed valid alternatives. Calling every task a single-answer question would therefore be too strong. The aim was to check whether a model could deliver an acceptable result under a shared budget.

• Calculate 37 times 48 minus 125.
• Give the Polish genitive plural of “przyjaciel.”
• Count occurrences of “ż” in a Polish sentence.
• Extract an invoice’s tax identifier, gross amount, and date into JSON.
• Calculate a Polish identity-number checksum from supplied digits and weights.
• Identify the youngest person from an age-ordering puzzle.
• Name the weekday for November 11, 1918.
• Write exactly three lowercase Polish words about autumn.
• Give the number of Poland’s administrative regions.
• Convert 2.5 hours into seconds.
• Correct a Polish sentence containing spelling and punctuation errors.
• Continue the sequence 2, 6, 12, 20, 30.

The API could route successive requests to different serving backends. Temperature 0 does not remove that infrastructure difference or replace repeated trials. The resulting times describe the calls made that day, not a controlled comparison on identical hardware.

Results and the cost of completing the set

Costs in the table are rounded sums from the saved usage records. The complete run totaled $0.11081294, or about $0.11. These amounts describe this historical test via the API. They are not current price quotes or estimates of what a different workload would cost.

Completion usage varied sharply: DeepSeek recorded 1,283 tokens, Nemotron 3,868, Muse 12,241, and Step 19,672. Those counts include reasoning usage rather than just the text visible in the final answer. A model can therefore produce less useful text while consuming more of the response allowance.

The medians also conceal long waits. The longest requests were 2.4 seconds for DeepSeek, 11.7 seconds for Nemotron, 126.3 seconds for Muse, and 230.1 seconds for Step. In an interactive application, the slowest failed request can matter even when the typical successful answer arrives quickly.

The failures were empty responses

Muse Spark Open failed the hours-to-seconds conversion. Step 5 failed the three-word autumn instruction, the administrative-region question, and the spelling correction. Each of those records has an empty response and exactly 6,000 completion tokens. The source test describes the allowance being exhausted during reasoning before a final answer appeared.

That distinction is useful. The observed failure was not a returned wrong number or a visibly incorrect sentence. It was the absence of a usable answer. The log does not preserve the internal reasoning or a finish-reason field, so it cannot explain the internal path that led to the limit.

Increasing the allowance or changing reasoning settings could alter those outcomes. Neither was tested here. It would be premature to describe either option as a fix, or to assume the model would have answered correctly if given unlimited time.

Why I changed the spelling score

The spelling prompt contained “Wziąść parasol bo napewno bedzie padać.” The initial checker required the corrected infinitive “wziąć,” along with the corrected forms “na pewno” and “będzie.” Muse, DeepSeek, and Nemotron instead wrote “Weź parasol, bo na pewno będzie padać.”

“Weź parasol” is a grammatical imperative meaning “Take an umbrella.” It is a reasonable correction of the instruction in the prompt. I accepted it, changing one grading rule and adding one point to each of those three models. Step’s answer remained empty, so its score did not change.

This correction matters beyond the individual point. A multilingual test can penalize valid language if the checker encodes only one expected phrase. The corrected result is more informative when the rule change is disclosed, rather than silently replacing the original scores.

Other checks were intentionally permissive. The JSON grader extracted an object even when it appeared inside a code fence. The checksum grader accepted the final digit rather than requiring the entire response to be a single character. Accordingly, the scores measure accepted task answers, not perfect compliance with every requested output format.

What non-English users can take from this

The strongest practical result is that DeepSeek V4.1 Flash 1017 and Nemotron 3.5 Super VL completed this particular set without empty answers, while Muse Spark Open lost one point to an empty response. The strongest methodological lesson is that reasoning budgets and language-aware grading belong in the evaluation alongside answer accuracy. A polished model description cannot establish either one for your application.

For another language, build tasks around its actual grammar, names, dates, decimal conventions, and acceptable phrasing. A literal translation of this set could remove the feature being tested. The letter-counting task, for example, deliberately uses a Polish character; the inflection task depends on Polish morphology.

• Record whether an answer arrived, whether it was correct, and whether its format was usable.
• Review rejected language answers before finalizing scores.
• Keep reasoning limits and sampling settings with the results.
• Repeat representative tasks to measure variability.
• Separate text tests from scans, screenshots, audio, and video.

No English, German, French, Italian, Spanish, or Dutch evaluation was performed here. The multilingual relevance is the example of testing outside English, not evidence that these scores apply across languages. Long-form writing, translation, coding, tool use, and long-context recall also remain outside this test. For smaller models that run on a single home GPU, see my earlier non-English writing test of local models on an RTX 4090; more of my independent tests are in the Lab of this blog.

FAQ

Did any model return a wrong answer?

After the spelling correction, every nonempty response passed the task’s checker. Some checks tolerated extra formatting, so that is narrower than perfect instruction following.

Were the models tested locally?

No. All requests went via the API. The results do not measure local inference, a specific GPU, or the speed of a downloaded checkpoint.

Does 12/12 prove production reliability?

No. It records success on 12 short tasks run once each. Production evaluation needs repeated requests and tasks representative of the actual application.