The biggest mistake we make with AI is trying to decide whether it’s good or bad. It’s a powerful, unreliable technology sitting inside an industry that rewards exaggeration. Sometimes that produces comedy. Sometimes it produces genuine danger. And sometimes it produces real progress.
The circus comes to town #
Every AI product launch follows the same script. Someone walks onto a stage and announces that work, education and possibly death have now been solved. The demo works perfectly, the valuation doubles, and nobody asks what happens when an actual customer tries to use it.
Revolutionary. Human-level. About to replace an entire department. Six months from transforming civilisation. Every launch, the same four claims - because hype gets funded and modesty doesn’t.
Regulators have started calling this out by name: “AI washing.” In 2024 the US Securities and Exchange Commission charged two investment advisers over false or misleading claims about their use of AI, and the Federal Trade Commission has gone after businesses selling supposed “AI lawyers” and automated money-making schemes that didn’t hold up.
Klarna is the cleanest example. The company said its AI assistant did the work of 700 customer-service staff. It later admitted it had leaned too hard on the cost-cutting story and started hiring humans back to fix the service quality it had traded away. The lesson isn’t that AI delivers no productivity gains - it’s that one good experiment quickly turns into a sweeping claim about the end of employment.
When it’s confidently wrong #
Generative AI’s defining trait: it can be completely wrong without losing an ounce of confidence. A normal computer error looks like an error. An AI mistake is beautifully written, neatly formatted, and backed by references that don’t exist.
Invented legal cases. Fake quotes. Company policies that were never written. Instructions that sound plausible and aren’t safe. Summaries that confidently reverse the point of the thing they’re summarising.
Air Canada found this out the expensive way - a customer relied on bereavement-fare information its chatbot invented, and the airline was ordered to pay up after unsuccessfully arguing it shouldn’t be held to what its own chatbot had said. Lawyers have made the same mistake in court: by mid-2025, fabricated case citations were turning up not just in legal filings but in judicial rulings.
The funny part is that AI makes things up. The less funny part is how quickly people stop checking, because the answer looks so convincing. The real failure is rarely the model alone - it’s an unreliable system, a pressured human, a badly designed workflow, and nobody left to verify anything.
When it stops being funny #
Skip the science-fiction version of AI danger. The real threats are already here: deepfake fraud, cloned voices, automated scams, political disinformation, biased decisions made at scale, and people handing important calls to systems they don’t understand.
Arup found out how far this has come. An employee in Hong Kong joined what looked like a routine video call with senior colleagues and authorised fifteen transfers totalling roughly £20 million. Every face and voice on that call was fake.
The frightening thing about AI isn’t that a computer became human. It’s that computers have gotten good enough at imitating humans to exploit the trust everything else depends on. AI didn’t invent lying, fraud or propaganda - it just gave them industrial machinery.
When it’s actually good #
Step away from chatbots and generated images for a second. Some of the least theatrical uses of AI are doing the most real work.
AlphaFold has predicted the structure of millions of proteins, handing researchers tools to chase down disease and treatments faster than was possible before - work now feeding into drug discovery, heart disease research and more resilient crops. In the NHS, AI-assisted scan analysis is helping clinicians spot blood clots faster, now running across more than 70 hospitals and getting stroke patients to specialist treatment sooner.
Earlier disease detection. Faster weather prediction. Better fraud detection. Administrative work quietly removed from public services so people can do the parts of the job that actually need a person. None of it tries to replace a human - it just gives one more reach.
The actual point #
Give it to an overexcited founder and it becomes a promise no technology could keep. Put it in an unchecked process and it becomes a confident generator of mistakes. Hand it to criminals and it becomes a machine for manufacturing trust and deception. Put it in the hands of a careful doctor, scientist or engineer, and it helps solve something that actually matters.
The future of AI won’t be decided by how smart the machines get. It’ll be decided by how smart we are about where, why and how we use them.
This is exactly why we started Repace.ai.
