Victoria Police say several people were arrested as the crowd "grew increasingly hostile" during the protest that began before 9am on Monday.
The union said only a minority of the protesters were union members.
"This crowd was heavily infiltrated by neo-Nazis and other right-wing extremist groups and it is clear that a minority of those who participated were actual union members," Dave Noonan, CFMEU national construction secretary, said in a statement on Monday evening, condemning the "far right extremist violence".
Hundreds of protesters stormed the Melbourne headquarters of the CFMEU, protesting against new mandatory vaccination rules for the building industry.
The protest escalated when two union officials, including Victorian construction branch secretary John Sekta, came outside the Elizabeth Street office to speak to protesters just before midday.
Mr Setka was met with boos and insults from the crowd, while some protesters hurled bottles.
"Please calm down. Can you at least give me the respect to talk? We're not the enemy, I don't know what you have heard," he told protesters.
"I have never, ever said I support mandatory vaccination."
Once Mr Setka went back inside, the protesters smashed a glass door to the building.
Later, bottles were thrown at the already smashed glass entrance doors, as those protecting the entry sought refuge indoors.
Protesters also turned against each other, with a number of small fights breaking out within the crowd.
'Freedom of choice'
The Victorian branch of the union released a statement just after 4pm, saying it has always support freedom of choice regarding vaccination.
"We are not going to be intimidated by outside extremists attempting to intimidate the union, by spreading misinformation and lies about the union's position.
"The CFMEU will always advocate for safety, jobs, and freedom of choice."
Speaking to the ABC on Monday afternoon, Mr Noonan said the violence was "disgraceful" and he urged the small number of union members who may have taken part in the protests to "walk away from extremists".
Chanting "f*** the jab", those protesting are against having to show proof of their vaccination to be allowed back on building sites, with some saying they would rather the entire construction industry be shut down.
There is currently a 25 per cent cap on the industry's workforce in Victoria, with tradies required to have had their first dose of a vaccine by this Thursday in order to be allowed back to work.
'Protests not smart, not safe'
Premier Daniel Andrews told reporters in Melbourne during his daily coronavirus update the protests were "not smart, they are not safe".
"Protests don't work. Getting vaccinated works, following the rules works. That's how you stay open, that's how you get open," Mr Andrews said.
Monday's rally comes after construction workers set up plastic chairs and tables in the middle of streets across Melbourne on Friday, protesting against lockdown restrictions that included shutting down tea rooms for morning breaks.
With SBS News.