Every week there is a breakthrough, an “AGI achieved” post, and a demo that looks like magic. Some of it is real progress. Most of it is a marketing department with a good editor.
The pace of AI announcements has outrun most people’s ability to sort signal from noise, which is exactly what a lot of the announcements are counting on. You do not need a computer science degree to avoid getting swept up. You need a few habits of skepticism, the same ones that protect you from any confident sales pitch, applied to a field that has learned to make its pitches look like science. Here is how to tell real AI progress from the hype wrapped around it.
The short version
Treat every AI demo as the best case a company chose to show you, not the typical one. Watch for the gap between flashy benchmark scores and how a tool actually performs day to day, and be suspicious of grand, vague claims like “build anything” or “AGI” with no clear definition. The fastest filter is to ask whether you can use the thing right now, whether anyone independent has tested it, and who profits from you believing the claim.
Table of Contents
The demo is not the product
The single most useful rule is that a demo shows you the best moment a company could produce, staged under conditions it controlled. The famous example is Google’s 2023 Gemini launch video, which looked like a person having a fluid, real-time spoken conversation with an AI. As IEEE Spectrum and others later detailed, the reality was edited: the exchange happened over text rather than voice, still images stood in for a live video feed, and the best responses were selected and narrated to feel effortless. The model was real, but the demo was a highlight reel. Assume every polished AI demo is doing some version of this until you can run the tool yourself.
Watch the benchmark-to-reality gap
Benchmark scores are the other place hype hides. A model can post state-of-the-art numbers on a test and still hallucinate, lose track of context, and frustrate you in ordinary use. Part of the problem is that models can be tuned to the tests, and benchmark questions sometimes leak into training data, which inflates the score without improving real ability. So a chart showing a new model beating the last one is a starting point, not proof. What matters is the gap between the benchmark and the boring question of whether it reliably does the specific thing you need.
The vague-claim tell
When a company cannot tell you specifically what its AI does or how, the vagueness is the message. “AI-powered,” “build anything,” and “AGI” are doing marketing work, not describing a capability. As Computer Weekly puts it, AI claims are cheap, and the real challenge is working out what is behind them. A genuine capability can be stated plainly: this tool transcribes audio at this accuracy, this model writes code that passes these tests. A claim that dissolves into buzzwords the moment you ask a concrete question is a claim to distrust.
Follow the incentives
Ask who benefits when you believe the hype. Grand AI announcements have a way of arriving right before a funding round, an earnings call, or a product a company needs to look ahead of. That does not make every big claim false, but it tells you to weigh the source. A splashy “we’ve achieved X” from a company that needs to raise money on the strength of X deserves more skepticism than a quiet result someone published for others to check and pick apart.
The questions that cut through it
When you see the next breakthrough claim, run it through a short filter:
- Can I actually use this now, or is it a demo and a waitlist? “Coming soon” is not a capability.
- Has anyone independent tested it, or is every number coming from the company selling it?
- Do they show failure cases, or only the wins? Honest tools admit what they get wrong.
- Is the claim specific and testable, or a vibe? “Build anything” is a vibe.
- Who profits from me believing this right now?
A claim that cannot survive those five questions is hype, however impressive the video looked.
What real progress actually looks like
Genuine advances tend to be quieter than the hype cycle suggests. They ship as something you can use, get poked at by independent researchers, come with acknowledged limits, and usually improve one specific thing reliably rather than promising everything at once. The instinct that serves you here is the same one that helps with any too-good online claim, which we wrote about in how to tell a real tech rumor from a viral hoax: check the source and its track record before you believe the headline. Reliability, boring as it sounds, is the real frontier, not the flashiest demo.
Frequently asked questions
How do I spot AI hype?
Treat demos as staged best cases, distrust benchmark scores that do not match real-world use, and be wary of vague claims like “AGI” or “build anything.” Then ask whether you can use the tool now, whether anyone independent verified it, and who profits from the claim.
Was the Google Gemini demo fake?
The model was real, but the 2023 demo video was edited to look more impressive than the live experience. The interaction was over text rather than voice, used still images instead of a live feed, and showed selected best responses, which made it feel smooth in a way the actual tool was not.
Are AI benchmarks reliable?
Only partly. Benchmarks are useful signals but can be gamed by tuning models to the tests, and test data sometimes leaks into training, inflating scores. A high benchmark does not guarantee a tool is reliable in everyday use, where hallucinations and errors still show up.
What does “AGI” actually mean?
There is no agreed definition, which is part of why the term is so easy to hype. Broadly it refers to AI with human-level general ability across tasks, but companies use it loosely, so an “AGI” claim with no specific, testable definition is more marketing than milestone.
Is AI overhyped?
Both things are true at once: the technology is genuinely useful and improving, and the marketing around it is heavily inflated. The skill is separating real, usable capability from announcements engineered to impress, using the red flags above.
What to do next
You do not have to be an expert to stay grounded about AI, just a little harder to impress. Assume the demo is a highlight reel, look past the benchmark to real reliability, distrust claims that cannot be stated plainly, and always ask who benefits. Do that and the endless stream of breakthroughs becomes much easier to read, with the genuine progress standing out precisely because it does not need the hype. For more on gadgets and consumer tech, browse The Other Stream’s Tech section.