Serco helps run immigration detention sites for the federal government.
Refugee Action Coalition spokesman Ian Rintoul says Queensland Health then told the ABF and Serco that the men couldn't be held at BITA because it's a quarantine facility.
He says the men were woken at 4am on Monday, bussed to the airport and flown to Melbourne.
"They were very hurriedly pushed out of Kangaroo Point last Friday and they just weren't ready," Mr Rintoul told AAP.
"We have been anticipating that they were going to be moved to Melbourne for some time, and it's just happening this morning."
Comment has been sought from Queensland Health and the Department of Home Affairs.
Mr Rintoul expects the men to be transferred to Melbourne's Park Hotel rather than the Melbourne Immigration Transit Accommodation facilty.
He said at least three of the men were told they would be handcuffed on the domestic flight, but he couldn't yet confirm that had happened.
"So they weren't in handcuffs when they left the detention centre in Brisbane, but they were told that we're going to be put in handcuffs while flying."
One asylum seeker told the advocate group: "We are not cattle, but we are treated like cattle."
Mr Rintoul says the 17 flying to Melbourne and another two remaining at BITA were not allowed to bring any possessions other than the clothes they were wearing with them.
He says all their clothes, toiletries, phone chargers, documents and other possessions are still in their rooms at the Kangaroo Point hotel.
Mr Rintoul says the men have had to borrow clothing from other BITA detainees.
"There is a long history of possessions being lost in the course of such transfers," he said.
"Every shift is traumatic. Most of these guys have lost everything more than once. On Manus their eviction from the old detention centre was violent.
"They should not have been shifted from Kangaroo Point hotel, they should have been released."