GPT-6 Astra Test: Non-English Writing, Code, 150K Context

👁 115 reads

On September 4th, GPT-6 Astra arrived on OpenRouter, the platform we’ve been using to test models for months. That same evening, we ran it through three sets of tasks, each with verifiable answers, and compared it against three competitors: Claude Fable 5.1 at the same price point, Claude Opus 5 at half the cost, and Gemini 3.8 Flash, which launched two days earlier at a fraction of the price.

The three test sets were: non-English language (six tasks covering declension, numerals, capitalization, participles, commas, and plain language - the same ones we’ve used since August, run in Polish), code (three PHP functions run against hidden test cases), and needle in haystack (five facts hidden in 150,000 characters of text). Everything at temperature 0.2, single run, costs measured from API billing. Upfront caveat: a single run isn’t a benchmark with hundreds of repetitions, but it shows how the models performed on our desk on day one.

Language Tasks: Gemini Got Perfect Score at Fraction of Cost

The biggest surprise doesn’t concern Astra. Gemini 3.8 Flash, a lower-tier model, was the only one to correct capitalization fully according to Polish norms: president of Warsaw, city office as unofficial name, and mister director in lowercase, Ministry of Digitization and Christmas in uppercase. Astra and Fable 5.1 left “Urzędzie Miasta” (City Office) capitalized, Opus 5 additionally capitalized “Pana Dyrektora” (Mister Director). These are exactly the errors most common in Polish official documents, so it’s hardly surprising the models replicate them.

Astra solved all six tasks and was the fastest of the four, averaging 6.5 seconds per response. For comparison, when Fable 5.1 premiered three days earlier, it was Anthropic’s model that hung on numerals; this time Opus 5 hung on the bureaucratic language task and remained stuck until connection timeout after four minutes. Freezing on short language tasks appears to be a characteristic of the Claude family, not a one-time incident.

At the same temperature, models aren’t repeatable. On the first measurement, Fable 5.1 wrote “urzędzie miasta“ in lowercase, this time uppercase. An assessment from a single run is therefore an assessment of that run, not the model in general.

Code: All 19 out of 19, Meaning Tasks Were Too Easy

Three PHP functions: slug from Polish text (with diacritic transliteration), correct noun form after numerals (file, files, of files, with traps like 12, 22, 101, 124), and merging overlapping intervals. We ran each answer against hidden tests: 19 assertions total.

Result: four models, 19 out of 19 for each. Honestly, this is a test failure, not a model success. Tasks that could still surprise a year ago (Polish numerals are a classic trap) are now solved without error even by the cheapest model in the lineup. Differences remained only in billing: Gemini 3.8 Flash $0.014 for three tasks, Opus 5 $0.039, Astra $0.040, Fable 5.1 $0.048. The next code test will need to be significantly harder to measure anything.

Needle in Haystack: All Found It, But Bills Differ Twentyfold

We inserted five fabricated facts (reservation code, frequency, aisle number, password with diacritics, shipping time) into 150,000 characters of Polish text at different positions, from the third to the ninety-sixth percentile of length, and asked five questions.

Everyone found everything, so qualitatively a tie. The interesting column is tokens: the same Polish text is 45,882 tokens for Astra and 43,350 for Gemini, but 73,063 for both Claude models. Anthropic’s tokenizer splits Polish into nearly 60 percent more pieces. At identical rates per million tokens, this means that for the same work in Polish, Fable 5.1 actually costs more than Astra, even though both price lists look the same. We’ve been tracking this “token tax” for a while in our cost calculator, and here you can see it in a single task: $0.58 for Astra, $0.73 for Fable, $0.03 for Gemini.

What This Means

First, Astra is fastest on these tasks and didn’t hang once, which after our experiences with the Claude family has practical weight for us. Second, in Polish it’s not better than cheaper models: it made the same error as Fable 5.1 and lost to Gemini 3.8 Flash, which cost twenty times less. Third, Astra’s advantages from official benchmarks - mathematics, abstract reasoning, and cybersecurity - don’t appear at all in daily editorial work in Polish, so this test doesn’t measure them and doesn’t pretend to.

The conclusion for someone writing and editing in Polish is the same one we drew when testing Fable 5.1: the most expensive tier doesn’t provide an advantage in this work, and among cheap models a candidate has emerged that does it without error. For someone who needs mathematics, agents, and long research tasks, this test says nothing and you need to wait for independent measurements. We’ve collected full model data in our GPT-6 Astra guide.

The method and raw responses from all models are recorded; if you want to see them or propose harder tasks for the next round, write to us.

// read moreGPT-6.1 Sol: Pricing, Availability & Non-English Writing TestNews7 Local AI Models on RTX 4090 vs Cloud: Non-English Writing TestBenchmarksGLM-5.3 Test: Non-English Writing, Pricing & ProvidersBenchmarks