The head of Tony Abbott's indigenous advisory council Warren Mundine says he has urged the prime minister to drop controversial plans to amend the Racial Discrimination Act.
But he says he won't walk away from the advisory council in protest.
Mr Mundine, an indigenous leader and former national Labor president, said he copped racial abuse nearly every day.
"When you let people off the chain in regard to bigotry then you start having problems," he told ABC radio on Wednesday.
Mr Mundine said society made it quite clear that racism and bigotry were unacceptable.
He said he had not considered standing down in protest as there was a big job to do in closing the gap, and getting people into jobs and education.
"I am not going to throw that out in an argument on this issue," he said.
"We have had conversations about it and our advice to the prime minister was that they should not be going down this track."
Labor, the Greens and some human rights groups fear amending the act will open the floodgates of racism and bigotry.
Mr Abbott says the government will maintain a "red light" against bigotry in a way that doesn't curb free speech.
Deputy Labor leader Tanya Plibersek said no one in Australia agreed with the government on this issue.
"It is a sad thing that one of the first priorities of this new government is to make it easier to racially abuse people," she told ABC radio.
Ms Plibersek said numerous community organisations had been contacting her and her colleagues to express their concerns.
Attorney-General George Brandis said this was at heart a free speech issue and limitations needed to be crafted as narrowly as possible.
He said racist abuse would not fall within the exemption and racial vilification would always capture the concept of Holocaust denial, as featured the case of Fredrick Toben.
"We want to protect racial minorities from the core concept of racism. The core concept of racism is the incitement of racial hatred or the causing of fear because of a person's race," he told ABC radio.
But Labor frontbencher Chris Bowen rejected the government's free speech argument, pointing out that the coalition had no plans to change defamation laws.
"The government seems to think that those restraints on freedom of speech are okay, but a requirement in a modern sophisticated civil society that public debate and discussion not include racial abuse is not okay, and it's okay to be a bigot," Mr Bowen told reporters.
Labor MP Ed Husic, the first Muslim to be appointed to the frontbench in the Australian parliament, said the racial discrimination laws had held the nation in "good stead for 20 years".
The changes were more than just symbolic, he said.
"It actually tries to encourage people to test the outer limits of how they can offend and cause division."
Labelling the proposed changes the "Andrew Bolt clause", Australian Greens leader Christine Milne said it would create a legal loophole to use offensive or inciteful language.
Independent senator Nick Xenophon said he would be talking to community groups to gauge their reaction, but Mr Mundine's opposition was worth noting.
"In a democratic country as ours you have a right to be a jerk and an idiot," he told reporters.
"But ... anything that incites hatred, that could potentially incite people to violence, is something that we must as a society do everything we can to prevent."
Scottish-born Labor senator Doug Cameron, who left his home country to escape bigotry, says Senator Brandis and the coalition should have a good look at themselves.
"They obviously don't understand what bigotry does to individuals, what bigotry does to communities."
