AI Will Write Its Own Tests. Who Marks Them?

Last time I said AI will get smart enough to create its own benchmarks. One question has been bugging me ever since: who marks that exam?

Picture a student who writes the exam, sits it, and then marks it. In school we'd call that cheating. With AI, I think we'll call it progress.

And once we can't mark it ourselves, I think there are only two examiners left. I don't fully trust either of them.

Prefer to watch? This post is also a short video.

Who holds the red pen

Start with what a benchmark actually is. It isn't just hard questions. It's questions plus an answer key, and right now we hold the key. We write the test, the model sits it, we check the answers. That's school. The teacher has the red pen.

Except we've already started handing the pen over, one step at a time.

Step one: the marking

Humanity's Last Exam, the one with 2,500 questions, is marked by another AI model: its leaderboard lists the judge model it uses. The key is still ours. The marking isn't. As far as I can tell, nobody minded. It's just faster.

Step two: the checking

This one is older than you'd think. In 1976, two mathematicians proved the four colour theorem: any map can be coloured with four colours so that no two neighbours match. They did it with a computer that checked more than a thousand cases, one by one. It took over a thousand hours. And some mathematicians wouldn't accept it, because no human could check it by hand.

Do you know how that argument ended? In 2005, the whole proof was checked again. By another program.

Read that back. We couldn't check the computer, so we got a second computer to check the first, and then we all relaxed.

Step three: the understanding

In 2016, AlphaGo played a move against Lee Sedol that most professionals wouldn't have considered. It didn't look like a move a human would play. Then it won the game. We only knew the move was good because of what happened after.

To be fair to us, Lee won the fourth game, with a move another professional called divine. One game out of five. That was ten years ago.

The key is slipping

So we've handed over the marking, the checking, and sometimes the understanding. What we kept was the key.

And now even that is slipping. FrontierMath now includes problems that are still unsolved by mathematicians. There is no answer key, because nobody has the answers. If an AI solves one, a computer checks the solution. In their words: "without needing to review AI proofs."

The only thing left is writing the questions. So imagine the day AI does that too.

The two examiners left

The first examiner is another AI. One model sets the test, a second one sits it, a third one marks it. That sounds rigorous, until you remember they all learned from much the same internet. Two students who copied from the same book will agree with each other all day. It doesn't make them right.

The second examiner is reality. The game is won. The bridge stands. The drug works. Reality never marks wrongly. But it marks late, and it only gives you the result, never the working. A bridge doesn't tell you which sum was wrong. It just falls. And some exams it only lets you sit once.

That's the part I can't shake. Benchmarks were how we kept up. Every rung on the ladder was one we built, so we could see how high it had climbed. Once AI builds the rungs, the ladder keeps going, and we're at the bottom looking up.

I already live like this

But here's the thing: I already live like this. I can't mark my doctor's exam. I can't check a pilot's working. I have never read the proof of the four colour theorem, and I never will. I don't trust the answers. I trust whoever checked them. None of us checks everything. That's how anything gets built.

So maybe our job changes. We stop marking the exam and start choosing the examiner. That's a much smaller job than the one we have now. I'm not sure it's a safer one.

And it leaves one question I can't answer. When the examiner is smarter than everyone in the room, how do you choose it? You'd need a test for that. And somebody would have to mark it.

Maybe the last benchmark we write won't be the hardest one. Just the last one we could mark.