Dubbed “Camp Sovereignty, the protestors have been camped in the gardens since the start of the Commonwealth Games last month.
Campaigning for indigenous rights, the group calling themselves Black GST (Genocide, Sovereignty, Treaty) lost a court battle earlier this week to keep the campsite.
The Supreme Court had given the group until two o'clock Thursday afternoon to remove their tents cars and caravans.
Prime Minister John Howard said last week that the Aboriginal tent embassy should be removed quickly before it takes root.
Even without the shelter of tents and caravans protestors are insisting they'll stay by the camp fire which they consider sacred.
They are allowed to keep their fire going for the next 30 days.