Members of the Aboriginal community have claimed there was not enough cultural consultation over the name and location of the Barangaroo development.
A fire at the Sydney site last week was the latest in a long list of troubles at the development, just two months after a young Aboriginal man fell to his death.
Some traditional owners, the Gadigal people, are claiming the site is cursed while other members of the community have voiced their opinions on social media websites.
The name of the development, selected as part of a competition from over 1600 entries, is taken from a woman believed to be the wife of great warrior Bennelong, a significant figure during the time Sydney was colonised.
Wiradjuri Poet Lorna Munro said the project was incorrectly named.
"I'd probably say that there is a negative energy in the area. Especially naming such a site after a woman who was against European development," she said.
A spokesperson for the Barangaroo Delivery Authority, who is also responsible for the development’s Indigenous Advisory board, defended the project in a statement.
"Barangaroo the woman will be honoured in the work that we do," the statement read.
"Aboriginal women have been involved in all the consultations we have held to date regarding the possible national centre of Indigenous art and culture, and in most cases have been the majority in the room.
"Women will continue to play a central role in ensuring that Barangaroo the woman is honoured."
But NITV has been told by sources close to the board that there are concerns there aren't enough Indigenous women involved, and that these concerns have been raised in those meetings.
Lorna Munro said she couldn't imagine the woman the development was named after would be happy.
"I would again say – look at the type of woman she was. She was a really strong, staunch black woman from that era," she said.
NITV attempted to contact a female Gadigal elder but was unable to find a suitable person before broadcast.
