Attorney General George Brandis insists the idea of a First Nations People body was never going to happen and is happy with the government's decision to ditch it.
A constitutionally-enshrined indigenous voice to parliament - a key recommendation of May's Uluru indigenous summit - was rejected by the Turnbull government last week.
Senator Brandis said it would have effectively been a third chamber in the parliament.
"This idea was a non-starter," he told Sky News on Sunday.
"I have no confidence that such radical proposal, if put to the people at a referendum, would have passed. I am sure it wouldn't have passed."
Manager of Opposition Business Tony Burke said the government's decision comes when it is now looking at the constitution after it lost two ministers because they were found to have dual citizenship and ineligible to stand in parliament.
He said it would be "outrageous" if there was a referendum to make life easier for politicians when it has rejected the requests for constitutional reform from the first Australians.
"To have a situation where the government decides - 'oh, that cause isn't popular enough' - and then to think that making life easier for politicians would be popular ... I can't see them going there," he told ABC television.
"If they do, I can imagine how loud the response would be from the Australian people."
