Riot police moved in to shut down scattered groups of people marching down Elizabeth and Collins streets in the CBD just before midday before the protest moved south.
Chanting "every day", hundreds - some still wearing high-visibility clothing like in days earlier - marched through the city to the Shrine of Remembrance war memorial on St Kilda Road.
Heavily armed police surrounded the Shrine on Wednesday afternoon, with officers slowing moving in on the mob, making arrests.
The stand-off lasted for more than three hours as police tried to negotiate with the mostly male and unmasked protesters to peacefully exit via St Kilda Road.
By 4:30pm, some of the mob started to disperse but dozens remained behind and became rowdy before police fired what appeared to be rubber bullets.
A flare was thrown in retaliation as the riot squad cleared the crowd and took control of the site, which was left strewn with broken bottles and rubbish.
Deputy Commissioner Ross Guenther said more than 200 arrests were made over the course of the day.
He told reporters on Wednesday evening it was "completely disrespectful" that the crowd ended up at the Shrine, "which is such hallowed ground in this city."
The Returned & Services League (RSL) Victoria also condemned those who it says effectively occupied the Shrine, saying the site is "sacred, not a space of protest".
"Under no circumstances, ever, should the Shrine be a place of protest," it said in a statement on Wednesday afternoon.
"If any individuals or groups choose to express their political views, positions or ideological theories in the grounds of the Shrine at any time, they are completely disrespecting the sanctity of this time-honoured space."
RSL Australia President Greg Melick called the protesters' actions a "disgrace to the nation" and said they "must be condemned in the strongest manner."
“The Shrine of Remembrance honours those who have served and made the ultimate sacrifice for their country and the protesters’ actions are nothing short of desecration of this revered site,” he said.
“Those involved in this lawless mob not only dishonour the men and women who fought and died for our country, they shame themselves, their families and all those involved in the protest."
Demonstrators have taken to the streets in the last three days to voice their distrust in the COVID-19 vaccine and anger at the state government.
While some attendees have been there to protest mandatory vaccinations for the construction industry, authorities say the demonstrators are not all union members or tradespeople.
“We have seen groups that contain people from the construction industry but they contain a range of other people - people are being encouraged to put on high-vis gear, to put on workwear, to come in under the cover of a crowd,” Victoria Police Chief Commissioner Shane Patton said on Wednesday morning.
CFMEU Victorian construction secretary John Setka previously blamed "neo-Nazis and right-wing extremists" for hijacking the events.
Victoria Police was on Wednesday granted temporary restricted airspace over Melbourne CBD by Australia's Civil Aviation Safety Authority, in order to protect officers' on the ground.
News choppers were grounded under the four-day order but will be allowed to take to the air in future, subject to approval and publishing footage on a 60-minute delay or after an operation has finished.
Daniel Andrews condemns demonstrations as Melbourne braces for more protests
In addition, police were permitted to use crowd control force against those trying to repeat the seven-hour "cat and mouse" game seen on Tuesday, when up to 2,000 protesters led police across the city and shut down the West Gate Bridge.
"I thought what we did today was very effective," Deputy Commissioner Guenther said.
Earlier on Wednesday, authorities had urged protesters planning to attend to stay home, with Mr Patton on Tuesday night warning "our tactics tomorrow will be different."
Police Minister Lisa Neville had called for the "thugs" to stay away from the city.
"If you're thinking about coming into the city today to cause violence and harm, just know that Victoria Police will deploy whatever tactics they need to in order to ensure that you are held accountable," she said.
'There for a fight'
Earlier, Premier Daniel Andrews launched a strident defence of construction workers, saying the people who have been turning up to "riot" were “there for a fight”.
“I have met hundreds and thousands of builders, hundreds and thousands of tradies who build this state … they're fine people, hardworking people. What we saw yesterday is an insult, an insult, to the vast, vast majority of tradies or people in the building industry who are not about wrecking, they're about building,” he said on Wednesday morning.
“They did not reflect and should not be seen to reflect an entire industry. That would be unfair. That would be wrong.”
The state government has shut down the construction industry for two weeks in metropolitan Melbourne, City of Ballarat, City of Greater Geelong, Surf Coast Shire and Mitchell Shire.
Mr Andrews said he was “gutted” to shut the industry for two weeks, but the move was forced due to the high number of COVID-19 cases in the sector.
“There are more coronavirus cases in construction than there are in aged care, more cases of coronavirus in the construction sector than there are patients with coronavirus in hospital across the whole hospital system,” he said.
Victoria on Wednesday reported 628 new COVID-19 cases - its highest daily tally in the current outbreak - and three deaths.