An Aboriginal protest group, campaigning as the Black GST (Genocide to end, Sovereignty acknowledged, Treaty to be made) set up a camp in Melbourne's Kings Domain during the Commonwealth Games and has since refused to leave.
The Victorian government won't intervene, saying it is a matter for the City of Melbourne.
Mr Howard says the camp should be removed as soon as possible. "It should be dealt with quickly. If it's left - and the Canberra experience is instructive - it stays," he told Southern Cross Radio.
"And I don't think that sort of thing should be allowed. I can understand why people didn't want to make an incident about it during the Commonwealth Games, that's just commonsense."
The group dubbed the games the "Stolenwealth Games" and replaced the "united by the moment" motto with "divided by the history".
Organisers had said they planned to pack up "Camp Sovereignty" on March 25, but have since declared a fire on the site as sacred.
Councillor David Wilson said a committee meeting last night acknowledged that the council was working with the group to "come to a peaceful solution".
Mr Howard said the camp could set back the cause of reconciliation. "These are the sorts of things that really set back the cause of reconciliation because it's the unacceptable face, in a way, of reconciliation," he said.