The government has also suspended the 47 other indigenous inspectors of Aboriginal heritage.
The action means there will be no inspectors to renew the 30-day order protecting the fire at a protest site in the Kings Domain when it expires on May 10.
A group of Aboriginal protesters set up the camp, dubbed Camp Sovereignty, during the Commonwealth Games to highlight what they said were issues of genocide and displacement.
Acting Victorian Premier John Thwaites has written to Prime Minister John Howard asking him to amend federal legislation under which the heritage inspectors are empowered.
"There is a widespread concern that there hasn't been adequate control over the granting of emergency declarations and that's why we're taking the steps that we're taking today," Mr Thwaites said.
"We don't want to see a situation where public confidence in our system of protecting Aboriginal heritage is undermined.
"That's why from now on the (Aboriginal Affairs) minister (Gavin Jennings) will be responsible, not inspectors."
Two weeks ago, the Supreme Court ordered the group dismantle the camp but allowed them to keep the fire going under the protection of the 30-day order.
The Victorian government is enacting its own legislation to cover indigenous cultural heritage, which now comes under federal law.
Until the bill, which is now before the Victorian parliament, becomes law, Mr Jennings will have sole power to issue emergency declarations.
"I'm waiting for any advice that comes to me that places the cultural heritage significance of that fire in that place," Mr Jennings said. "Up until now I haven't seen any evidence."