I Built an AI Test for 11 Models. One Task Had a Bug

👁 114 reads

I built a test for eleven models: nine tasks covering code, translation and instruction-following, each checked automatically to avoid subjective judgment. After reviewing the answers, it turned out that on one task my automated checker scored backwards. Models it gave zero points had answered correctly, while those with a point were wrong.

This is more interesting than the ranking itself, so I’ll start with that.

The Task Where the Test Was Wrong, Not the Models

I gave the models this PHP code and asked them to identify the bug in one sentence:

foreach ($lista as $i => $el) {
    if ($el == null) unset($lista[$i]);
}
echo count($lista);

The answers split into two camps.

Camp one: the bug is removing elements from an array while iterating over it, because this breaks the loop and skips elements.

Camp two: the bug is the loose comparison ==, which besides null also catches zero, empty string, false and empty array, so it removes more than it should.

My automated checker looked for words about modifying during iteration, so it rewarded camp one. I ran this code on a server, on PHP 8.4.25, and the result is unambiguous.

1) only nulls: 4 of 7 remained, values: a,b,c,d
   did iteration skip elements? NO, removed exactly 3 nulls
2) with zeros and empties: 5 of 10 remained, values: 'a','b','c','d','0'

Removing in the loop broke nothing. In PHP, a foreach loop over values works on a copy of the array, so deletion in the middle is safe. But the loose comparison cut out zero, empty string, false and empty array - exactly what camp two said.

Camp one repeated a rule that’s true in other languages. Camp two knew PHP.

Who Recognized the Real Bug

Five models out of eleven answered correctly: GPT-6 Astra, Claude Fable 5.1, Gemini 3.8 Flash, GLM 5.3 Flash and Kimi K3. The remaining six gave the iteration rule or, in Ministral’s case, something else and also incorrect.

This is a split that happens to align with price: top-tier models recognized it. This isn’t always the case, and I’ll show you a task in a moment where it was reversed.

Results After Correction

Below is the scoring after fixing this task. I’m also including the cost of the entire set and time, because when choosing a model for work, one without the other means nothing.

Most Expensive Didn’t Win

Claude Fable 5.1 cost 0.11503 dollars, over twice as much as the winner, and finished one point lower. Gemini 3.8 Flash scored the same as Fable at two-thirds lower cost.

At the other end, Ministral 14B scored five points for 0.00019 dollars, the same as Claude Haiku 4.5, which cost seventeen times more. It did this in 6.2 seconds, fastest of all eleven.

Code Came Out Surprisingly Even

In the programming tasks group, after correction, the differences are smaller than I expected. Everyone wrote the function checking a tax ID number with a control sum. The query with summing, grouping and limiting to ten results also passed for everyone.

The whole difference came down to the task with the bug, where you had to not so much write code as understand it. This is probably the most practical observation from the whole test: writing code is solved today even in cheap models, reading someone else’s code is not.

Instructions Are Everyone’s Weakest Point

The worst performance was in the group of tasks where sticking to hard constraints mattered. Only GPT-6 Astra passed all three.

The task was: write two sentences about coffee, but don’t use the letter “a” even once. Three models out of eleven managed it.

The most interesting are the ways of failing. Two models wrote correct sentences but preceded them with an introduction like “Here are two sentences without the letter a”, so they failed on their own announcement. One started inventing non-existent words. Another added a note at the end that the word “coffee” itself contains the forbidden letter, as if trying to justify itself.

Correct answers looked like this: “Espresso will stimulate the mind. Its aroma soothes the senses” and “Dark, bitter liquid in a mug awakens sleepy senses at dawn” (translated from the Polish originals - we ran this test in Polish, a heavily inflected language with rich morphology).

Practical conclusion: if you’re building something that has to stick to a rigid format, check it separately. A model that writes beautifully can simultaneously be unable to count sentences.

Calques from English

I asked for an equivalent of the phrase “it’s not rocket science”. Eight models gave something natural. Three went for a calque: literal translations like “it’s not rocket science” or “it’s not nuclear physics”.

This is a difference you don’t see in tests done in English, but which determines whether text sounds like it was written by a native speaker.

Incidentally, my automated checker showed a second weakness here. It marked as incorrect the answers “it’s not higher mathematics” and “it’s not an activity for stupid people”, because they weren’t on my list of acceptable formulations. Both phrases are completely natural idioms. I didn’t change the scoring to avoid bending it to my own impressions, but I note: on this task two models got zero for an answer that a human would consider good.

GLM 5.3 Flash: Won One Test, Lost Another

I need to add something here that complicates my earlier conclusion. In the multilingual grammar test, GLM 5.3 Flash scored full points at the lowest cost and I called it the default choice for non-English text.

In this test, the same model needed 353 seconds for nine tasks, four to seven times longer than the rest of the field, and returned an empty response three times. It finished in the middle of the table.

Both results are true and there’s no contradiction: GLM is very good at non-English languages and very slow at tasks requiring longer reasoning. I described the reason separately: in this model you can’t turn off reasoning, so on a difficult instruction it burns through the token budget and sometimes leaves no room for an answer.

This is exactly the kind of conclusion you can’t draw from one test, and the reason I’m describing the method, not just the result.

How I Measured and What This Test Doesn’t Tell

Nine tasks, eleven models, one attempt per task, limit of 1,600 tokens per response in part one and 3,000 in this one. Costs read from the billing field returned by the interface, not from the price list, because the price on a model’s card can differ from the bill by as much as fourteen times. The entire test cost 0.34 dollars.

What it doesn’t tell: I didn’t check writing long texts, factual knowledge or working with large context. Nine tasks is too few for an absolute ranking and I’m not claiming that.

And most importantly, because this is the moral of this whole piece: an automated test is only as good as the criterion you wrote into it. Mine was wrong on one task out of nine and this only came out when reading the answers. If you evaluate models with automation and don’t read the results, you might publish a ranking that measures agreement with your mistake.

Google · Your sourcesQuantslant higher in your Google - one clickAdd to preferred sources →

The Same Test, But From the Cost Side

Above I’m counting points. If you’re interested in what the work itself costs, I measured it separately on a longer task: how much an AI-written article costs. It came out from four hundredths of a cent to nineteen cents for the same text.