LMArena: How to Read the AI Model Ranking and Where It Fails

👁 126 reads

LMArena (rebranded as Arena in January 2026, at arena.ai) is the most frequently cited AI model ranking. It doesn’t measure test performance - it collects votes from people who ask a question to two anonymous models and pick the better answer. Over 8 million such votes produce a leaderboard with a score for each model. We explain where these numbers come from, how to read confidence intervals, and why first place in Arena doesn’t mean the model will solve your task best.

What LMArena Is and Who Runs It

The project began in 2023 as Chatbot Arena at UC Berkeley, in the LMSYS group. Creators Wei-Lin Chiang and Anastasios Angelopoulos described the method in a March 2024 paper, when the platform had 240 thousand votes. In April 2025 the project became Arena Intelligence Inc., in May 2025 it raised $100 million (at a $600 million valuation), and in January 2026 another $150 million at a $1.7 billion valuation. On January 28, 2026, the LMArena brand changed to Arena.

The company makes money from paid evaluations for AI labs (AI Evaluations service since September 2025) and according to its own June 2026 data has $100 million in annual revenue. We wrote about this in our article on Arena’s revenue. The site reports over 354 million sessions, and labs test models there before launch under codenames: OpenAI tested GPT-5 as “summit”.

How the Ranking Is Created

The mechanism has three steps:

• Battle. In Battle mode you enter a question, get two answers from two anonymous models, and vote: A better, B better, tie, or both bad. Model names are revealed only after you vote.
• Bradley-Terry model. From millions of win/loss pairs, the algorithm calculates a single number for each model, such that the score difference translates to the probability of winning a battle. It’s the same family of methods as Elo in chess, just computed on the entire dataset at once rather than sequentially.
• Confidence interval. Each score has a ± value. A model with 3 thousand votes has ±11, a model with 70 thousand has ±4. If two models’ intervals overlap, their order in the table is random.

The leaderboard also has a “Style Control” toggle. When enabled, the algorithm removes the influence of answer length, number of headings and lists, to measure content rather than packaging. This matters because people more often vote for longer answers with bullet points.

Leaderboards: Text, Image, Video, Code, WebDev

Arena runs over a dozen separate rankings: Text, Agent, WebDev (building a webpage from description), Image-to-WebDev, Vision (questions about images), Document, Search, Text-to-Image, Image Edit, Text-to-Video, Image-to-Video, and Video Edit. Each leaderboard has its own votes and its own rankings: the best model in text may be mid-pack in WebDev. Columns to read:

Example from the text leaderboard on September 13, 2026: Meta’s muse-spark-1.2 has 1500 points at ±11 with 3,227 votes, while claude-opus-4-6 has 1497 at ±3 with 75,878 votes. On paper Meta is higher, statistically it’s a tie.

Where the Ranking Misleads

Style versus substance. Voters see answers for a few seconds. They reward confident tone, length, and formatting, rarely checking facts. That’s why Arena doesn’t measure hallucinations; separate tests do that, like our hallucination test in Polish.

Non-English languages. The leaderboard has a language filter, but there aren’t enough battles in most languages for narrow confidence intervals. The overall score reflects English and Chinese. If you work in other languages, check our comparison of models in Polish.

Lab gaming. In April 2025, Meta submitted an experimental version of Llama 4 Maverick to Arena, different from what it later released publicly. The paper “The Leaderboard Illusion” (Cohere, AI2, Princeton, Stanford) calculated that before Llama 4’s launch, Meta privately tested 27 text variants (43 including vision), and Google and OpenAI received 19.2% and 20.4% of all platform data respectively, while 83 open models together got 29.7%. Two identical models submitted under different names received scores differing by 17 points. Arena responded on May 9, 2025, defending pre-release testing and changing rules for publishing withdrawn variant scores.

How to Use Arena in Practice

• Go to arena.ai, choose Side by Side mode and select two specific models you’re considering. Paste your own task: a client email, contract excerpt, function to fix.
• Repeat with 5-10 different tasks. One battle is anecdote, ten is signal.
• On the leaderboard, enable Style Control and category filter (e.g. “Coding” or “Longer Query”), because a model good at short questions may lose on long ones.
• Check the Votes column. A new model with 3k votes may drop several places in a month.

We publish a monthly leaderboard review, the latest being the July 2026 ranking. The basics of how models work are explained in our article what is an LLM.

Frequently Asked Questions

Is LMArena free?

Yes. Battles, side-by-side comparison mode, and direct chat with a chosen model are free. Without logging in you get a limited number of messages, after logging in more. Only the evaluation service for companies is paid.

Can you test in other languages?

Yes, models respond in any language, and your vote goes into the pool. However, rankings filtered by non-English languages have wide confidence intervals because there are few votes.

How often does the ranking change?

The leaderboard is recalculated every few days, new models usually appear on launch day or a few days after. The date of the last update is shown below the table.

Does the Arena score tell you which model is “smartest”?

No. It tells you which answers people chose more often in a blind test. It’s a measure of preference, not correctness. Our editorial view: for choosing a model for work, Arena is a good starting point and a bad ending point.

// read moreWhich AI Model for Work in 2026? Choose by Task, Not RankingGuidesMicrosoft-Decision-1: Ranking, Pricing and Test ResultsBenchmarksAI Detectors: How to Read Results and Check Text ProperlyGuides

Kinetyra