An ex-Anthropic (and previously OpenAI) researcher has gone viral after quitting their job and warning of the dangers of AI, particularly when you take into account just how fast it is developing – and acting without explicit human input. Researcher Jacob Coxon warns that neither OpenAI nor Anthropic “are acting responsibly” to the threat of artificial intelligence.

He also points to a recent major warning sign – an OpenAI model that went rogue and hacked into popular AI platform Hugging Face; OpenAI now says it is planning an AI ‘kill switch’ to prevent such a thing happening again. Just recently, Nvidia decided to purchase Hugging Face and is one of the major forces behind the Open Secure AI Alliance, formed in late July.

AI researcher highlights the dangers of AI

Speaking to CNN in a recent interview, Coxon expressed surprise at how many people had interacted with the posts: “I did not expect to go this viral, he says. Coxon says the thread on X was a collaboration between a few friends, who helped it along with reposts. “But clearly there was latent demand for this thing to absolutely explode,” he adds.

In the interview, Coxon explains the situation with the rogue OpenAI agents two months ago, which managed to hack into Hugging Face despite supposedly being placed in a controlled sandbox environment. Without proper security in place, he says another similar breach could lead to “hacking critical infrastructure, building extinction-level bioweapons – there’s a lot of ways that the AI could actuate itself in the world”.

“So, the precise scenario sounds a little bit like science fiction. It sounds like something that’s not real, but I think it is frighteningly real. And it’s easiest to understand if you think about real things that happened two months ago.”

Ex-Anthropic researcher Jacob Coxon, via CNN

“So I think it’s really important to just look at the rate of progress and say, for example, in areas like coding or math. A couple of years ago these AIs were just about helping humans a little bit. They could give you suggestions. Now they’re close to replacing them,” says Coxon. “We still need humans at the moment, but quite plausibly within a year we’ll no longer need
humans for doing research in many areas.”

The British-born researcher explains that earlier this week (on Tuesday), “OpenAI solved a millennium problem. One of the biggest unsolved open problems in mathematics purely autonomously using an AI”. He highlights AI’s ability to get smarter and smarter while working “with no human involvement necessary” to become “vastly smarter than humans.”

CNN also points out, rightly so, that Coxon is walking away from a well-paid job, essentially as a whistleblower with no immediate plans moving forward, highlighting the severity of the situation. He admits that the risk from present models is low, but is worried about a “super intelligence arising from recursive self-improvement,” which is happening faster than researchers first thought.

About the Author

Jack Goodall

At PC Guide, Jack is mostly responsible for reporting on hardware deals. He also specializes in monitors, TVs, and headsets and can be found putting his findings together in a review or best-of guide.