Reverend Garry Dronfield said he receives multiple harassing phone calls each time he raises the flag outside the Sylvania Uniting Church.
“People ringing up saying they're offended. Saying they want the Australian flag up not the Aboriginal flag," Mr Dronfield told NITV News.
Mr Dronfield flies the flag ahead of a monthly Indigenous focused church service, which he said is an important event for the local Aboriginal community.
"Sitting around having a yarn…We have a meal together and we play instruments of Aboriginal type music with the didgeridoo, clap sticks and boomerangs,” Mr Dronfield said.
“It sort of brings the whole cultural part of us together."
And he says it's always open for everyone to attend.
“We have not only Aboriginal people come on our Aboriginal service, we have Tongan people, Fijian people, anybody that wants to come,” Mr Dronfield said.
Murri Elder Frank Logan is a regular attendee and he said it provides an essential space for communal healing.
“Most of our people are more like ones who sit back and sort of suffer in isolation:"
"And a church as a communal community also can help and can also understand where they are coming from because it is about sharing," Mr Logan said.
Reverend Dronfield said he puts some of the harassment down to the church's location in the Sutherland Shire.
"It's not a cheap place to rent or buy a house so you get a sort of demographic of people that live here. And that demographic, some are really supportive and others aren't,” he said.
But for those who call in to complain about the flag he has a simple message.
"Come along and have a look what we do. Please support us. You don't have to love us, but you can support us," he said.