Where's the Safety Inspector?

Nobody has to prove that a toaster is safe before they’re allowed to sell it. Well, actually, somebody does.

There are rules for toasters. Rules for elevators. Rules for bridges. Rules for electrical wiring. Even playground equipment has standards, because we’ve collectively agreed that children shouldn’t be launched into the shrubbery because someone skipped a safety inspection.

This is how civilization works. If you build something powerful enough to hurt people, there are laws, standards, inspectors, liability, and, when all else fails, the possibility that someone in authority can tell you to stop.

But not for AI.

The biggest AI companies are building increasingly powerful autonomous systems while, by their own employees’ and leaders’ accounts, acknowledging that they don’t fully know how to control them. Some are openly discussing serious risks of serious harm using phrases like "catastrophic or existential risks to humanity.”

That should make us pause.

If an elevator company said, “We’ve built a new elevator that can make its own decisions, we’re not entirely sure what it’ll do, and there’s a decent chance someone could get hurt,” I suspect the regulatory response would be fairly brisk.

Nobody would say, “Interesting. Maybe let’s put 40 people in one and see how the experiment goes.”

But with AI, it’s build the thing, release the thing, watch millions of people interact with the thing, and hope everything turn out OK.

I’m not arguing against AI. I’m just suggesting we need oversight and regulation.

If a company is building systems powerful enough to act autonomously and potentially cause serious harm, it should have to demonstrate that those systems are safe enough to deploy. Test them. Stress them. Find the failure modes. Fix what breaks. Bring in people who don’t work for the company to kick the tires.

And if the answer is, “We can’t reliably control this yet,” then perhaps the product isn’t ready for prime time.