Cheryl Lawdorn and her partner have been flying the flag over their Beulah property in the state's northwest for several weeks, because of her German ancestry, she told The Age newspaper, which published photos of the flag.
Victorain Premier Daniel Andrews said, "nothing justifies that (flying the nazi flag) whatsoever and it should come down".
The flag has a black swastika in the middle on a white background, encircled, with Nazi symbols on red backgrounds in each corner.
"The people who are displaying that despicable flag, this is disgusting - it is absolutely disgusting behaviour," Mr Andrews told reporters on Tuesday.
"If there's any decency in that household, they will take that flag down immediately."
Victoria's parliament is currently running an inquiry into racial vilification with a focus on increased anti-semitic behaviour seen recently.
