Mr Abbott campaigned on the plan to repeal Section 18-C of the Act, which prohibits remarks that offend others on the grounds of race or ethnicity.
The Government has argued Section 18-C limits free speech.
But council chairman Warren Mundine says a strong majority of the council feels the section is needed.
"We've got to be very vigilant. And I think it's pretty much a weak argument they've put forward about freedom of speech. It's a philosophical stance they're taking and not dealing with the reality of what happens out on the ground with people."
Opposition frontbencher Chris Bowen says the Prime Minister should take Mr Mundine's advice seriously.
"We agree that the Racial Discrimination Act as it's currently formulated is appropriate, we don't support changing it and yes, if Mr Mundine has put that view then that would be something I would hope that the Prime Minister would seriously consider."