Benchmark

A benchmark is a standardized test for comparing models: a set of tasks in math, general knowledge, programming, or language understanding, where each model receives a score. Thanks to benchmarks, we can say that model X handles code better than Y. Results should be taken with a grain of salt: popular tests eventually "leak" into training data, and producers optimize models for the metrics. That's why alongside classic benchmarks, rankings based on blind human evaluations and private tests are growing - and the best benchmark remains your own task.