Indigenous organisations say the cashless debit card is discriminatory, punitive, and ineffective, following a Senate inquiry backing the government's plan to expand the controversial scheme.
The scheme sees 80 per cent of people's Centrelink payments moved onto a card to only be used for things the government deems 'essential'.
A parliamentary inquiry recommended that a bill, which would see the card made permanent for 25,000 people in the Northern Territory and Cape York in Queensland, should pass through parliament.
People in Ceduna in South Australia, Western Australia's East Kimberley and Goldfields regions, and Bundaberg and Hervey Bay in Queensland would also be permanently moved onto the cards.
Amnesty International Australia's Indigenous rights advisor Rodney Dillon said the scheme was "at best paternalistic, at worst, simply racist".
"I think this is another act of superiority over our people," he told NITV News.
"Picking on the most vulnerable people in this country and deciding what's best for them. This is a bit like the stolen wages saga or the stolen children saga.
"It's about being able to voice your authority and use your authority over a group of people, without consulting with them or without any good recommendation that it works."
Mr Dillon said any income management policies the government makes should be based on consultation with the communities they affect.
"These people are making these decisions for our people without consulting with them, without seeing how it works and without being out there and having a look at the effects," he said.
'Paternalistic and punitive'
National Aboriginal and Torres Strait Islander Legal Service (NATSILS) Executive officer Roxanne Moore said the scheme "clearly targets" Indigenous people.
"You can see that from the trial sites - Ceduna, the East Kimberley and Goldfields in Western Australia, and Hervey Bay and Bundaberg in Queensland - they all have significant populations of Aboriginal and Torres Strait Islander people," she said.
"Human rights organisations have raised concerns about racial discrimination. It is paternalistic and punitive, and not only that, but it is in conflict with the Closing the Gap agreement.
"There is no evidence that this has an impact on the issues it aims to affect."
Ms Moore said NATSILS opposes the recommendation that the bill to expand the use of the card should be passed.
"This is clearly harmful," she said.
"The impacts on people - there is shame and stigma associated with these cards, and outages so people may be without food or medication, or may be late to pay their rent because they can't access their money.
"We need to put Aboriginal people in the lead of solutions in their own communities."