In brief
- A senior Anthropic researcher says there is a greater than 10% chance AI could kill all humans within a decade.
- Experts say the numbers are subjective but the point is there is a risk, however small or subjective.
As AI systems grow more powerful, some of their creators are attempting to estimate the odds of the technology's darkest possible outcomes: human extinction.
Evan Hubinger, who leads alignment research at AI company Anthropic, recently said he believed there was a greater than 10 per cent chance AI "could kill all humans" within the next decade.
His comments followed the resignation of Anthropic researcher Jacob Coxon, who reportedly warned AI companies were racing towards self-improving superintelligence without adequate safeguards.
Hubinger is far from alone. Several prominent AI researchers have publicly estimated the likelihood that advanced AI could cause catastrophic harm.
Nobel Prize-winning computer scientist Geoffrey Hinton has previously put the chance of AI causing human extinction within the next 30 years at between 10 and 20 per cent.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
A newly released survey of 1,580 researchers who had published at leading AI events found respondents assigned an average 18 per cent chance that future AI advances could lead to human extinction or similar permanent or disempowerment of humanity. The median estimate was 10 per cent.
But the increasingly alarming figures raise an obvious question: how do you calculate the probability of something that has never happened?
Where do the numbers come from?
Dr Chaitanya Joshi, a senior lecturer in statistics at Adelaide University, said the figures should not be confused with probabilities calculated from historical data.
"The answer is no, it's a subjective probability," she told SBS News.
"In fact, it is a type of situation where we will probably never have data to estimate a probability."
Joshi said that meant people should consider not only an expert's estimate, but also the reliability of their expertise and the level of uncertainty surrounding it.
Associate Professor Michael Noetel from the University of Queensland, who studies how AI experts assess catastrophic risks, said the subjective nature of these estimates did not however render them meaningless.
"Whenever we're dealing with risk, the kind of right way to manage risk is to try to get some estimate of the chances it could happen," he told SBS News.
"It matters if it's one in a million or a one in 10 chance of catastrophe."
Noetel said experienced forecasters can develop skills in probabilistic thinking, even if there is no known "true answer" against which AI extinction forecasts can be tested.
More than a 'gut feeling'?
Joshi said attaching a precise figure to an unprecedented event could give people the impression there was more scientific evidence behind it than actually existed.
Rather than treating 10 per cent as an exact figure, she said researchers could also quantify the uncertainty surrounding the estimate.
"Maybe 0.1 is your median estimate, but actually it could be as low as zero, it could be as high as 0.3," she said.
Meanwhile, researchers are also trying to make expert forecasts more rigorous.
Noetel for example, co-authored a study released this year which used a structured process known as the Delphi method to survey 272 AI experts from 37 countries.
Under current trajectories, the experts judged 18 of 24 categories of AI risk as having at least a 10 per cent probability of causing catastrophic outcomes within five years. Noetel said structured forecasting methods encourage experts to weigh competing arguments rather than rely solely on instinct.
So, what could actually go catastrophically wrong?
Noetel said the experts involved in his research broadly identified three clusters of risk.
The first was the deliberate misuse of AI, including cybercrime, terrorism or assistance in developing dangerous biological weapons.
The second was the possibility that increasingly capable AI systems could develop faster than humans' ability to govern or control them.
The third was an extreme concentration of economic and political power in the hands of the company or country controlling the most capable systems.
"The simplest way I would try to condense this down is losing control of the AI, someone misusing it, or the whole world economy being run by one country or one company," he said.
But exactly how a hypothetical superintelligent AI could cause human extinction was actually impossible to know, Noetel said.
He compared the problem to playing chess against one of the world's best players: a less capable opponent may be confident they will lose without being able to predict exactly how.
In the interim, researchers are looking for indicators that advanced AI could become increasingly difficult to control.
One is whether AI systems become capable of accelerating their own development by helping to design more powerful successors. Another is whether researchers can continue to understand and monitor how increasingly sophisticated models arrive at their decisions.
"If those two things continue, then the model is going to get faster and faster and faster, and they're going to be less easy for us to be able to control and monitor," Noetel said.
So how seriously should we take the numbers?
For Noetel, uncertainty about whether the true risk is 1, 10 or 20 per cent should not be taken as evidence that there is no risk at all.
"Some people think it's uncertain, therefore it will be okay... That is crazy" he said.
He compared humanity's situation to speeding through thick fog on a bus while some of the people at the wheel warn there may be a cliff ahead.
"That should not make you feel comfort that there is uncertainty," he said.
"It shouldn't lead to complacency."
For the latest from SBS News, download our app and subscribe to our newsletter.
