When Melbourne police first knocked on Cliff and Josef's door in the middle of the night, they ignored it.
Unknown to them, a crane at a neighbouring building site was in danger of collapse after being "compromised" by high winds and emergency crews were rushing to evacuate residents.
There was "a lot of noise" before the men were forced to leave their Richmond apartment around midnight on Monday, and they may not be allowed to return for days.
Work Safe has not ruled out laying charges over the problematic crane, which was damaged in winds at the Lennox Street construction site.
Broken pieces of crane could be seen hanging from the structure on Tuesday and there are fears it could topple as damaging winds continue to lash the city.
"The weather has placed it into a compromised position," WorkSafe's head of hazardous industries, Michael Coffey, told reporters.
Up to 80 people within 100sq m of the site were evacuated to Richmond Town Hall, according to Victoria Police's Rebecca Draper-Schultheiss.
Earlier, the Metropolitan Fire Brigade said 300 people had been forced to leave their homes.
Residents are unlikely to be able to go back until Wednesday at the earliest.
"We're a little bit dictated to on how quickly we can move because of the inclement weather," Inspector Draper-Schultheiss said.
Michael Clark from Clark Cranes earlier told 3AW the machine's arm appeared not to have been left in a position which would have allowed it to move freely in windy conditions.
"If the gib is left up too high, the wind can grab it and push it backwards, which it has done, unfortunately," he said.
"At some sites people complain about it overhanging a little bit and so builders lift it up that little bit more to appease the residents."
When asked about whether charges could be laid over the incident, WorkSafe's head of hazardous industries, Michael Coffey, said "this is a matter we are actively making an inquiry on".
"The disruption to local residents and traffic is regrettable but the safety of workers and the public must be our top consideration," he said.
Cliff said he and Josef were told they could be kept out of their apartment until Thursday and were staying with friends in the meantime.
"It's frustrating. I'm also glad that nobody got hurt because I heard something fall hours beforehand," he told AAP.
Under the "best possible scenario" the crane is expected to be dismantled on Wednesday as long as it's safe to do so, Mr Coffey said.
Strong winds continued on Tuesday, at one point snapping police tape cordoning-off the building site.
It is likely that the recovery of the crane will involve the use of two mobile cranes.
