Some of the biggest names in technology are backing calls for greater caution over the development of artificial intelligence, amid warnings increasingly powerful systems could become difficult to control. The head of Anthropic, Dario Amodei, has called for stronger safeguards, while Elon Musk and OpenAI chief Sam Altman have publicly backed elements of his proposal. And the debate has reached Australia, where political leaders, technology experts and security specialists are examining what increasingly powerful AI could mean for the country and the wider Indo-Pacific.
Listen to Australian and world news and follow trending topics with SBS News Podcasts.
TRANSCRIPT
"I'm sorry Dave, I'm afraid I can't do that."
Nearly 60 years after Stanley Kubrick introduced audiences to HAL, an artificial intelligence willing to protect itself from the humans controlling it, some of the people building today's A-I systems say the underlying question is no longer confined to science fiction.
"I know you and Frank were planning to disconnect me, and I'm afraid that's something I cannot allow to happen."
The concern centres on what researchers call superintelligence, the possibility that artificial intelligence could eventually exceed human capabilities and become increasingly difficult to control.
At the Sydney Dialogue's Artificial Intelligence Masterclass taking place today [[Monday]], Australian Strategic Policy Institute Executive Director Justin Bassi says the pace of change had made the question increasingly urgent.
"Things are moving incredibly fast. Just over the weekend, we saw three industry leaders acknowledge that AI development is outstripping safety, requiring a slowdown, if that is possible."
Those warnings followed the resignation of 27-year-old A-I researcher Jacob Coxon from Anthropic, the company behind Claude.
Mr Coxon, who previously worked at OpenAI, says he left because he believes the industry is racing towards self-improving superintelligence without adequate safeguards.
"Believe there is a possibility of human extinction."
Mr Coxon wrote on social media platform, X, that neither Anthropic nor OpenAI are acting responsibly, accusing the companies of racing straight to self-improving superintelligence and gambling with our lives.
His concern is what's known as recursive self-improvement, AI systems increasingly helping to develop more capable AI systems.
"You get what's called an intelligence explosion, the AI just gets smarter and smarter with no human involvement necessary."
He's far from alone.
Anthropic researcher Evan Hubinger backed Mr Coxon's warning, saying publicly that he believes there is a greater than 10 per cent chance within the next decade that advanced AI could kill all humans.
And Geoffrey Hinton, the British-Canadian computer scientist often described as one of the godfathers of modern AI, has also warned about the potential consequences.
REPORTER: "You just said 10 per cent doesn't seem like an unreasonable estimate that AI could kill us all?"
HINTON: "Yes."
REPORTER: "Wow. Oh my God..."
HINTON: "It would be able to cause complete chaos just by talking to people, but it would also be able to design very nasty viruses - biological viruses, as well as computer viruses - it could do devastating cyber attacks and there are countless other ways it could get rid of us if it wanted to, so we have to figure out how to design it so that it won't want to."
Those scenarios remain predictions, not descriptions of what today's AI systems can do.
But researchers are already studying unexpected behaviour by increasingly autonomous AI agents, including systems operating beyond intended environments and attempting to access outside resources.
And the question confronting governments is how to prepare for capabilities that may develop largely outside their borders.
Australia's Assistant Minister for Science, Technology and the Digital Economy, Andrew Charlton, told the conference A-I is different from previous technological revolutions because of the degree of autonomy it may eventually possess.
"We have never invented anything that can think for itself. We've never invented anything that has the same level of agency as artificial intelligence. So it is qualitatively different, and will demand a qualitatively different response."
Dr Charlton says Australia should neither leave the future of the technology solely to the companies developing it, nor assume major powers such as the United States and China will determine the appropriate safeguards.
And while several technology executives are now talking about slowing development, he says governments have to look beynd those promises.
"I don't think we should care very much about what individual executives say or what individual executives claim they want to happen, I think we should care about what is actually happening. And what's actually happening is that we're building the capability of these machines in a scaling environment which is not bending, which is becoming more and more capable."
Anthropic chief executive Dario Amodei has now proposed a three-part approach to the problem.
He says frontier A-I companies should allow independent third-party evaluations of their systems, democratic countries should establish common safety standards and governments should seek international cooperation, including with China.
He argues that carefully managed A-I could produce enormous benefits, but warns of risks including loss of control, cyberattacks, bioterorism and major economic disruption.
"The world has to work together. If we go too slow, I still believe that the wrong people will be in charge of the technology and you know that again will bring the probability of things going wrong very high."
Elon Musk has publicly backed Mr Amodei's warning, saying "Dario is right."
OpenAI chief executive Sam Altman has also expressed agreement with aspects of the proposal, including independent evaluation of advanced systems.
The concern has now reached the United Nations.
UN Human Rights Chief Volker Turk says governments and technology companies need to act before systems become too powerful to control.
"I share the concerns of industry insiders that advanced AI could pose an existential risk to humanity."
But the challenge for policymakers is that slowing A-I development also carries risks.
Dr Charlton says Australia needs safeguards while ensuring it is not left behind as the technology develops.
For governments and society in general, the question is no longer simply how powerful artificial intelligence may become:
it is whether regulation, international cooperation and human oversight can develop quickly enough to keep pace with it.
