He said he believed they were far-right activists after seeing the men’s clothing and appearance through the peephole in his door.
“They demanded to see me. I’ve got threats from the far-right before," he said.
Mr Gibson quickly moved with his partner to the back of the house.
“They started smashing on the door, they ripped the grill off the window at the front and they smashed the front window out with a chair," he said.
“We were really worried that they were trying to force entry and get into the house and hurt us."
Neighbours were immediately on the scene, Mr Gibson said, calling out to ensure everyone in the house was safe, and giving the all-clear when the men had left the premises.
Mr Gibson said it had been a "pretty scary few hours" and he was making plans to fortify his home.
Mr Gibson rose to prominence last July after organising a rally in Sydney that supported the Black Lives Matter movement. He was arrested at the rally after police had banned the event due to COVID-19 concerns. He is also a researcher at the University of Technology, Sydney and part of the Solidarity socialist movement.
He said his involvement in social justice causes had attracted abuse from far-right groups in the past, but until now, they'd been limited to social media or email.
“Last year when the Black Lives Matter rallies were happening … I was getting death threats at that time but no one’s ever come to my house. This was an escalation.”
NSW Police has confirmed detectives are investigating reports of “malicious damage”.
A spokesperson said officers arrived at the house after a report came in shortly after 7:30, and they then established a crime scene.
‘Far-right emboldened, openly organising’
The presence of far-right groups and Nazi symbols have become more frequent in Australia in recent months, including at 'freedom' rallies across the country.
In October, congregants at a Brisbane synagogue were confronted by the sight of a Nazi flag flying from a nearby apartment window.
At the time, Brisbane Lord Mayor Adrian Schrinner labelled the display as "sickening" and "pure evil", and said current laws were “inadequate”.
He said the incident would "likely to be classified as nothing more than a low-level public nuisance".
Queensland MP Ali King blamed vaccine conspiracy theorists for a Nazi swastika symbol painted on her office last month.
The Australian government last month named a neo-Nazi group known as The Base as a terrorist group following advice from security agencies.
"There is no place in Australia for their hateful ideologies," Home Affairs Minister Karen Andrews said at the time.
"We know that there are individuals actively watching what is happening in Australia. There are people here who have the intent and the capability to do us harm."
The Victorian government has announced it is working on new laws to ban the public display of Nazi symbols, in what would be a national first.
Mr Gibson said the far-right and neo-Nazis were “capitalising on anger that’s out there” as a result of the coronavirus pandemic, job insecurity and vaccine mandates to “set their own agenda”.
“It’s a very dangerous situation when the far-right are as emboldened as they are now and openly organising in the context of these 'freedom' marches," he said.
“It does mean that there will be more attacks on people of colour, on Aboriginal people, even white people like myself who actually organise for justice and against racism. It’s a very dangerous situation for everyone.”
George Newhouse, CEO of the National Justice Project, called on authorities to take prompt steps to bring the perpetrators of the attack on Mr Gibson's house to justice.
“Governments around Australia have failed to take the threat of violent racist and militant groups seriously and this is the end result. In the US, we have seen lives lost because of the demonisation of the Black Lives Matter movement by the right,” he said in a statement.
“It’s time for our leaders to heed the calls for racial justice and protect those who are calling for change.”