Why AI Benchmarks Are Fake

ARC-AGI: What’s the Hype?

ARC-AGI, the Abstraction and Reasoning Corpus for Artificial General Intelligence, is a 2019 benchmark by François Chollet. It aims to test whether an AI can solve novel problems it has never seen before, the purest form of out-of-distribution generalization.

OpenAI’s o3: The Latest Shiny Toy

OpenAI’s newest model, o3, reportedly scored 87.5% on ARC-AGI, surpassing average human performance. Impressive, sure. But one score on one test does not equal omniscience.

The Cynics: “Benchmarks Are Overrated”

Critics argue models can “game” benchmarks by exploiting patterns in the test format. Like memorizing an answer key, you can ace the exam, for instance get 100 percent, without truly understanding the subject.

The Fans: “Benchmarks Drive Progress”

Supporters say benchmarks provide targets and a shared yardstick to track progress over time. Without measurement, you are flying blind. It is like trying to get fit without ever stepping on a scale.

The Realist’s View: A Balanced Take

No single test captures the full scope of intelligence. A sudoku wizard might still panic in a crowded airport. ARC-AGI is useful to probe specific capabilities, but it is not the be-all, end-all measure of sentience.

Conclusion: Beyond the Flashy Scores

o3 made waves, but that does not mean AGI is around the corner. Real-world intelligence is about adaptability, creativity, and handling messy edge cases. Benchmarks are useful, just not sufficient. If we want real progress, we should evaluate how models perform in unpredictable, high-stakes, real contexts.

How This Relates to Lucido and How I Can Help

The debate around ARC-AGI and benchmarks fits Lucido’s focus on practical intelligence. Beyond headline scores, I help teams design balanced evaluations, realistic test scenarios, and clear storytelling for stakeholders, so improvements are both measurable and meaningful.