'A line has to be drawn': The trade-off we're making without really noticing

Artificial intelligence is increasingly used by police departments across Australia, with experts warning that lines must be drawn over the technology's use, invoking the dystopian world depicted in George Orwell's novel Nineteen Eighty-Four as a potential future to avoid.

Police departments are starting to trial the use of AI technology for "various forms of surveillance" and "predictive policing", professor Toby Walsh, chief scientist at the UNSW AI Institute, told SBS News.

In WA, police have been trialling AI facial recognition software to scan pedestrians on the street and compare their faces against watchlists. In the trial's first week, 130,000 people were scanned, leading to 18 arrests.

Walsh said that while most people would be "pleased" with the result, he believes such public surveillance is "an erosion of the things that we used to value".

"The question is, how much are we willing to trade those values for the benefits that the technology is bringing?"

Overseas, the rise of AI in surveillance technology has become an increasingly contested topic.

In the United States, citizens have been taking action against the surveillance cameras of the AI company Flock Safety, which has tens of thousands of devices across the country. Videos of people destroying the cameras have gone viral on social media.

Tom Sulston, head of policy at the advocacy group Digital Rights Watch, told SBS News that, while Flock cameras are not currently used outside of the US, "plenty" of companies in Australia are doing "similar things".

Palantir, the US data analytics firm contracted by military and US immigration forces, and Cellebrite, an Israeli digital forensics company working with law enforcement, both have tens of millions of dollars in government and private contracts in Australia.

"We should be concerned," Sulston said. "We have a right to go out in public without being surveilled, without having our biometric data uploaded to the cloud, and without it being used for purposes we don't really know."

Both have said that governments across the country need to rapidly turn their focus to expanding individual privacy laws, particularly as wearable surveillance technology like smart glasses become more widespread.

"We do have some legislation that covers public surveillance, but the more companies that get into that market, the more pressure there will be to relax them," Sulston said.

Walsh argues that a line between privacy and security "has to be drawn" as AI has made previously impossible levels of surveillance mundane.

"The one thing that George Orwell got wrong: it wasn't people watching people. It was computers, particularly artificial intelligence, watching people," he said.

What do Flock cameras do?

Flock's 120,000 cameras across the US mainly record licence plate and vehicle information for a searchable database.

The company states it is not in the business of mass surveillance, only in targeted crime prevention and response. Data is encrypted, held by the company for 30 days, and access to it is only available to customers who are logged.

Sulston notes that the information captured can be layered with other datasets from companies like Palantir to "produce really rich surveillance data of practically everyone that goes out in public".

"They have been doing that, and they've been using that to create pre-crime systems that try to predict where crimes will happen and inform police of where they think a crime is going to happen," he said.

Such systems, like many AI tools, frequently demonstrate bias in their responses and serve to further disempower "traditionally overpoliced people, whether that's immigrants or non-white people," Sulston said.

"There are real civil liberties issues".

While the company has yet to make any international moves, Flock chief executive Garrett Langley referenced Australia's relatively centralised police — spread across just eight forces — as an example of a potentially lucrative overseas market.

AI policing in Australia

While Australia doesn't have anything matching the Flock network's capabilities, police forces are increasingly using AI in their operations.

NSW Police notes on its website that they are primarily interested in using AI for "the automatic processing of high volumes of data for the identification of relevant laws, procedures, and precedents" but that they are also looking into "novel ways in which artificial intelligence can be used to prevent/reduce crime."

At present, high-volume data processing such as mobile phone detection or seatbelt offences are managed by machine learning throughout Australia. Victorian police note that, while AI flags potential offences, humans review and issue fines.

Walsh argues that a slow normalisation of AI into everyday policing could lead to social changes in which police develop a "tendency to defer to the machine".

More broadly, he argues that the law and its human application has been superseded in this area by AI, which operates in a much stricter fashion.

"I'm not so sure that we actually have the public's buy-in," he said.

In the US, the Washington Post reported that at least 50 law enforcement officers had been charged with misusing data access — including Flock information — to monitor romantic partners or ex-partners. In Australia, dozens of police officers have been charged with similar offences.

"We have to be really careful about the data we collect and the oversight that we have of the people who have access to that data," Sulston said.

'We need the government to step in'

The use of AI in surveillance technology is not limited to policing.

In 2024, Coles signed a three-year contract with Palantir to optimise its workforce and streamline its supply chains across 840 supermarkets in Australia.

Bunnings uses AI-driven facial recognition technology to monitor customers in their stores.

Sulston said the rising popularity of AI is supercharging already existing privacy concerns and that present laws "aren't really working for us".

"We need the government to step in, prioritise privacy reform, and ensure that the laws available to us can handle these new threats that we're facing," he said.

Walsh argues that the country needs to handle the rise of AI with caution and should learn lessons from the retroactive regulation of social media.

"We need transparency, he said. "The public needs to be assured of where and when it's being used and not being used, and under what circumstances and under what limits".

"There's still the social licence here, but I suspect if we're not careful, it will disappear very rapidly."

For the latest from SBS News, download our app and subscribe to our newsletter.