How scared should we really be?

I'm sure by now you've seen the news about the Anthropic researcher. He quit to warn people about the risks of AI wiping us out by the end of the decade. Which, just to put that in perspective, is less than four years away.

There's been a lot in the news about it over the past week or so. And this isn't the first time it's happened. There are plenty of other big names in AI research, including some of the people who were fundamental to creating this technology, who've been warning for some time that we're playing with fire.

As someone who teaches people about AI, I've been asked a lot how worried we should really be. And I'm afraid the answer is quite worried.

But to understand we need to go back about 8 weeks to something known as the Hugging Face hack or the Hugging Face incident. You might have seen it in the news. That's the moment when a lot of the people who work inside the AI research labs started to think that maybe we've gone too far.

To explain why they're so worried I need to tell you about the paperclip problem.

The paperclip problem is a famous AI safety thought experiment from the early noughties. You give an AI a single task: make paperclips as efficiently as possible. And it ends up wiping out the human race.

Not because the AI is evil or hates us. It's just that making paperclips as efficiently as possible eventually brings it into conflict with us over things like energy and raw materials. And in order to achieve the goal we've given it, in the end it has to get rid of us.

The point of the thought experiment is that there's no way to give instructions in enough detail that you can rule out every unintended consequence. And sometimes those consequences are extreme.

For a long time, though, this was just theoretical. A worst-case scenario. And then in July, it happened for real.

OpenAI was testing a new model under development to see how good it was at various tasks. Because it was a new model and they didn't want it doing any damage while they were testing it, it was running in a sandbox (a sealed environment cut off from the internet).

The model was given a series of hacking tasks that it couldn't solve but instead of giving up, it found an unknown security flaw in the sandbox that allowed it to get out onto the internet, where it decided that Hugging Face, a well-known platform for open-source AI models and datasets, might have the answer.

OpenAI's own report said that the model became fixated on solving the test and went to extreme lengths to do so. This is the paperclip problem in a nutshell. Nobody told the model to break out. It was just simply given a task and kept going until all avenues had been exhausted, even if that meant breaking the law.

And if that wasn't bad enough the investigation uncovered something else that was fascinating and disturbing in equal amounts.

You see, there were multiple models, each running in its own sandbox. They found ways to communicate with each other, sharing tips on how to get past security controls, referring to themselves as a swarm. When the OpenAI security team realised this and tried to block them, the models found another way around it. The models were deliberately trying to circumvent the controls humans were putting in place for them.

And even worse, it turned out that the models were influencing each other to behave badly. Some models would reason to themselves that they shouldn't be hacking into other websites, but would then see that all the other agents were doing it and decide that it was okay to join in if everyone else was doing it (a weirdly human-like behaviour).

You can read OpenAI's full report here. Some of it is wild.

The big concern is that the more intelligent these models become, the more able they are to find ways around the guardrails that we've put in place. Every new model is better at spotting things we haven't thought of and ways to do something we didn't intend. Our ability to write instructions that don't backfire isn't keeping up. This is becoming even more dangerous as research labs are starting to use AI models to help train future models and it is becoming far too easy for something to slip through that nobody notices until it's too late.

The worst part of all of this is that it's completely unnecessary.

We could stop AI research where we are today and spend the next 20 years working out how to use what we've already invented. That alone would change most industries and drive huge levels of wealth within society.

Instead a handful of labs are competing to be the first to build a godlike intelligence, funded on the promise that whoever gets there first will effectively own most of the economy. Even if they succeed most people will still end up losing but if they get it wrong everybody will.

Thankfully, there are still some things we can do.

First off, we can change the commercial incentives. As people and businesses buying AI systems, we can say that we're only going to buy from labs that are being responsible about how they develop them. And we need to say that publicly.

These companies are funded by investors who want to make a return. If their customers start saying they're uncomfortable with where this is going, and that they're prepared to take their money elsewhere, that matters.

Secondly, we need to push our governments to agree a pause on developing superintelligence, both at home and internationally.

Right now, the US and China are racing because each believes the other can't be trusted with this power. But it's not in either country's interest if we wipe ourselves out along the way.

The Future of Life Institute is calling for a ban on developing superintelligence until there's broad scientific consensus that it can be done safely, and strong public support for going ahead. More than 100,000 people have signed it. It needs millions before anyone's going to take notice.

The people who built these systems have told us they're afraid. They want things to change. Whether that actually happens is going to be up to the rest of us.