Benchmarks Won't Measure AGI. They'll Lead to It.
We have a lot of benchmarks these days: FrontierMath, Humanity's Last Exam, GDPval-AA and so on. But what do they actually mean when it comes to measuring AGI? Human intelligence isn't something a single test can quantify: we can perform even in scenarios we've never seen. So which benchmark could tell us whether AI has reached human level?
Prefer to watch? This post is also a short video.
What would we even measure?
Take IQ. It covers language, maths and pattern recognition, and it has to be re-normed every decade or two because scores kept rising. Meanwhile, we have a near-24/7 stream coming in through our eyes, and we're making decisions and analysing every moment. Our ability to live, compete, know and learn: is that what we'd have to measure to compare AI with a human?
A doomsday scale
On the Kardashev scale, we're at about 0.73. Will there be an AGI scale that says AI is 0.X of a human? If I were to give it a name: a doomsday scale. Someone has already tried: a 2025 paper scored GPT-4 at 27% and GPT-5 at 57%.
Benchmarks that lead to AGI
My point of view is different. There can't be a single benchmark that measures the doomsday scale. It won't be benchmarks that measure AGI; it will be benchmarks that lead to AGI.
We keep coming up with them: some for maths, some for multilingual understanding, some for pattern recognition, others for knowledge. Companies are benchmaxxing their newest models, and each generation is better than the last on most of them. People are creative, and we don't really understand what it takes to be human. Everyone has their own opinion, and every opinion becomes another benchmark.
So with N benchmarks covering 0.X of human capacity, every year it converges towards 1, to the point where AI itself becomes smart enough to create its own benchmarks.
The evolution we lived through, we're watching happen again right now: from a single-celled textual language model to a multi-cellular, multi-modal, agentic AI.