The flag - which photos show has a black swastika in the middle and Nazi symbols in each corner - was subsequently taken down by police.
Fuelled by shock that someone in Beulah could fly a Nazi a flag, about 60 locals and past residents met near their local pool on Saturday to highlight their welcoming spirit.
They held the Aboriginal flag, LGBTIQIA+ flag and the flags of as many nations as they could find, with South Korea, Afghanistan, India and Canada also among those represented.
The event was organised by the Beulah Historical, Learning and Progress Association only within the past two days.
"We've strung them all amongst the trees...we had lots and lots of flags flying," the association's secretary Barb Moore told AAP.
"Everyone was just so happy to come and put a happy slant on the thing, and just wanted to get together to show that we're a great town here and it's all good."
Ms Moore said the town had been in "absolute shock" that anyone could put a Nazi flag up.
"We just wanted to show look, that's not us."
She understands the couple who raised the flag have left town for the time being.
"I think they sort of worked out that it's not acceptable and I don't think they'll be doing it again at all."
Nobody tried to interrupt Saturday's event.
Beulah federal MP Dr Anne Webster and state MP Ali Cupper were among attendees, with both commending residents for their response to the incident.
"You've shown Australia and you've really shown the world what you need to do to as a community to stand up for what's right," Dr Webster told the crowd.
The demonstrating locals signed a petition calling on the state government to ban public displays of the Nazi flag and hope to rally more signatures before sending it off.
Chair of civil rights group the Anti-Defamation Commission Dr Dvir Abramovich said such a ban was the right way to go.
"We all have to be unified on this issue, and to fight the growing threat of bigotry and extremism which is a real threat to our way of life," he said.
Victorian Premier Daniel Andrews earlier this week described the flag as "disgusting".
Victoria's parliament is currently running an inquiry into racial vilification with a focus on increased anti-semitic behaviour seen recently.
Beulah had a population of 329 at the time of 2016 census.