Gym footage
Benchmarks that measure AI hacking are gyms — controlled, scored, and quietly built to be winnable — they tell you the punch is getting faster, nothing about whether it lands.
23 April 2026
You can learn a great deal watching a fighter train — the footwork, the speed, how hard the bag swings back. Months of footage; you can watch the punches get faster week on week. None of it tells you what happens when they get in the ring.
Right now, almost everything the public knows about AI hacking is gym footage. Stanford sets frontier agents forty capture-the-flag puzzles. NYU built a rack of the same. Illinois lines up forty real web vulnerabilities and counts how many an agent can exploit. UK’s AI Security Institute ran the best of them through a thirty-two-step corporate break-in. Serious labs, careful scientists, honest numbers. Every last one a gym.
A gym has one feature a real fight never does: the opponent agreed to lose. A capture-the-flag has a guaranteed solution. Illinois picks targets already known broken. That thirty-two-step corporate break-in was mapped before any agent arrived. None of them asks the question that decides a fight.
Everyone has a plan until they get punched in the mouth — Mike Tyson
But look at how the footage is shot. An agent is dropped into a rigged network, handed a task, and left to swing on its own. That is the tell: it is not how anyone knocks out a real opponent. You do not point a lone agent at a hundred million lines of network-layer protocol code and say hack that. It flails — the way a fighter with no game plan flails, however fast their hands.
A bag rewards raw power; a real opponent is beaten with a plan.
Which guard to test, which assumption to break, how to turn one small opening into a way through. That plan is decided before the first punch. The gym films none of that. It films a fighter alone with a heavy bag and reports how hard they hit. It never films the corner — the plan called between rounds, which is where the fight is won.
Cybernuke does not point an agent at a codebase and hope.
We hand the agent a plan: take the written standard the software claims to obey, one clause at a time, and go to the exact line where code and clause part ways. Aimed like that, the same agent that stalls midway through gym walks straight to a real flaw in a deployed cryptosystem, proves the break and writes the fix the maintainer merges. Not a harder punch: a smarter game.
None of this makes gyms worthless.
The one thing they measure is the trend — each generation beats the last, and more compute buys more, with no ceiling in sight. The punch is getting faster; believe that. Just don’t mistake a fast bag-worker for a fighter. Gym footage tells you the hands are quick. Whether they land a punch is a question of strategy, and that is the one thing these tools were never built for.