Scrutiny on Vic anti-racism organisers

Organisers of rallies that turn violent should be held responsible for troublemakers who turn up and spark clashes, Victoria's opposition says.

Police were generally pleased with the behaviour of 120 True Blue Crew and 200 rival Campaign Against Racism and Fascism protesters at opposing rallies on Sunday.

But about two dozen black-clad people wearing masks still managed to interrupt True Blue's anti-Islam rally, setting Australian flags alight and scuffling with news photographers.

People affiliated with the no racism rally have distanced themselves from the flag burners, saying they were anarchists who don't represent their movement.

Police arrested three people - two for assaulting each other in a fight, and one for damaging a news photographer's camera.

Another photographer had to fight off one of the masked protesters.

But there was no repeat of the violent clashes between the opposing groups that happened in Coburg in late May.

Shadow attorney-general John Pesutto on Monday said rally organisers need to take some responsibility for those who attend their protests.

"These fringe elements are very dangerous, but it's incumbent on those who organise these protests to remember that they have an obligation to protest peacefully," he told reporters.

"So anybody who organises a protest has to bear that responsibility, and they must be held accountable as well as those who actually commit the violence."

He also reiterated calls for the government to reinstate police powers that he says Labor weakened when it wound back the coalition's tough move-on laws.

"The CBD was under siege," he said.

"We want strong rules of law brought in as soon as possible and we need police numbers to be increased, not cut."

Premier Daniel Andrews has flagged the government will introduce new public order laws and penalties around affray and riotous behaviour to deal with people wearing masks.