Malcolm Turnbull is urging almost 600 men holed up inside the mothballed immigration detention facility on Manus Island to comply with Papua-New Guinea law and move to alternative accommodation.
A showdown is looming at the centre with PNG authorities warning they will use force to remove barricaded refugees and asylum seekers.
The men believe it's safer to remain in the compound than risk being attacked by locals at new facilities near the main township of Lorengau.
PNG officials entered the compound on Friday, emptying water tanks, destroying makeshift water-collectors, tearing down shelters and again advising the men to leave.
The prime minister insists the new facilities are fully-equipped and resourced.
"They need food, water, security, healthcare, they have all that spare," Mr Turnbull told reporters in the Vietnamese city of Da Nang where is attending the APEC summit.
They also needed to comply with PNG law, as did activists including Greens senator Nick McKim who were encouraging the men to stay put.
"They are acting in a way contrary to the interests of those refugees and other residents," Mr Turnbull said.
Law Council of Australia president Fiona McLeod said Mr Turnbull should heed a report by the UN Human Rights Committee which raised concerns about the Manus shutdown "without adequate arrangements for long-term viable relocation solutions".
"The committee's report is a clear and definitive statement about steps the government needs to take to address the human rights concerns identified," Ms McLeod said in a statement on Friday.
Manus Island authorities have issued a written warning to those still in the compound, saying force will be used to clear the centre.
It's expected PNG police could launch an operation as early as Saturday.
Iranian refugee Behrouz Boochani said the use of force was unacceptable.
"We are not doing anything wrong, we are only resisting peacefully," he told AAP.
"We are asking again for freedom in a safe third country."
