They say the family were taken to a nearby airport and flown to Melbourne, and are currently being held, facing deportation.
The council also claimed the family was not coping well with the conditions.
A spokesman for the council, Aran Mylvaganam, told SBS News the family left Sri Lanka to escape a horrible situation, including torture and the unexplained disappearance of family members.
“Successive Sri Lankan governments have lied about their human rights records, and they continued on abusing the rights of Tamils,” he said.
“Sri Lanka is in a state of fear for Tamils and other minority groups on the island - it's not a safe place."
The family had lived in Biloela for four years, with Nadesalingam and Priya arriving separately by boat before meeting and marrying in Australia.
The federal government said the family's asylum status had been under investigation for "many years", and was ultimately found to "not to meet Australia's protection obligations".
In a statement, the Department of Home Affairs said: "foreign nationals who do not hold a valid visa and who have exhausted all outstanding avenues to remain in Australia are expected to depart voluntarily to their country of citizenship".
It said, "those unwilling to depart voluntarily will be subject to detention and removal from Australia".
But it denies the removal of the family of four was done in anything other than a safe and orderly way.
According to the council, Dharuniga and Kopiga travelled in the same van as their mother, though they weren't allowed to sit with her.
The parents claim they were also pressured into signing documents agreeing to their "voluntary removal".
Mr Mylvaganam alleges the family were misled over their visa status and were expecting to have their bridging visa renewed.
“That's why although they knew their visa was to expire on the 4th of March, they waited, which is the case with many asylum seekers - when their visa expires, the immigration (case managers) renews it after the expiry date and they thought that that was the case in their situation as well,” he said.
Simon Bruck, a senior solicitor with the Refugee Advice and Casework Service, said many people aren’t able to present their arguments for why they are claiming asylum, and don’t have the English skills or legal know-how to properly navigate the system.
“People are going through this process unrepresented, without the expert legal advice that they require to get through a complex process,” he said.
“It’s an unfair process.”
Family friend Angela Fredericks has started a petition on change.org to return what she describes as a "beautiful, gentle family" back to Biloela.
Having already garnered more than 18,000 signatures, Ms Fredericks has pleaded with Home Affairs Minister Peter Dutton to let them go.
She said they are very much "wanted and welcome" in the community – which she says is not the case in Sri Lanka.
"All their family are either dead, killed, or else... they've left as refugees and are in other countries,” she said.
“So we're returning them to nothing when here in Australia they have jobs, they have friends, they've got support, they're welcomed here - why would we send them back?"