Mr Dutton said only four asylum seekers were left on the island - down from a high of about 1,300 in 2014 - and that they too would likely be gone in the coming weeks.
He framed it as a Coalition victory, fixing Labor's "humanitarian disaster" - but it is a claim disputed by those who were once detained there.
"We are still in Papua New Guinea ... We are not safe and we are still in limbo," one asylum seeker who was recently moved off the island told SBS News.
How did we get here?
Manus Island in Papua New Guinea, along with the Nauru detention centre, were first opened to house asylum seekers who came to Australia by boat in 2001 under then-prime minister John Howard's "Pacific Solution".
They were closed in line with a Labor election promise in 2007, triggering an influx of asylum seekers making the treacherous journey by sea. More than 1,200 people drowned trying to get to Australia, sparking a major rethink of the Labor government's border policy.
Then-prime minister Julia Gillard announced the government would resume offshore processing in 2012.
Under the policy, which remains in place today, any asylum seeker who arrives by boat has no chance of being allowed to settle in Australia.
While the tough approach successfully deterred boat arrivals, almost immediately Manus Island and Nauru's facilities attracted widespread international criticism.
"Australia's offshore processing centres are unsustainable, inhumane and contrary to human rights," the UN High Commissioner for Human Rights declared.
PNG grew weary of hosting the facilities, culminating with a Supreme Court ruling finding the detention to be illegal.
In 2016, the Coalition announced a refugee swap deal with the US, which has resulted in 630 people leaving Manus Island and Nauru.
The regional processing centre on Manus Island officially closed in 2017 to comply with the court order but many asylum seekers were transferred to other Australian-funded facilities on the island.
In August, the remaining asylum seekers on Manus Island who had not been resettled in the US or returned to their home country were made an offer.
A notice appeared on the island from Papua New Guinea's Immigration and Citizenship Authority. It offered a move to the capital of Port Moresby with the promise they would "continue to receive services like those currently available to them".
In the months that followed, all but four of the asylum seekers boarded planes to the capital.
But, some say, the prospect of their permanent resettlement looks as far away as ever.
'We are human beings like everyone'
Shaminda Kanapathi fled his native Sri Lanka and tried to come to Australia by boat in 2013.
After being intercepted, he was sent to detention facilities on Manus Island, where he spent the following six years.
"We were kept there to send a message for those who were waiting to come to Australia – this is what will happen to you if you come," he told SBS News.
Mr Kanapathi said he was surprised when the remaining asylum seekers on the island were offered the move to Port Moresby a few months ago.
"They didn't give us any reason … We didn't want to move out of Manus Island but [we thought] if we refused, they would force us."
Mr Kanapathi was taken to a hotel in Port Moresby but said many asylum seekers are scared to leave its grounds.
He said some friends "have been robbed, they've been attacked by local people".
"I want people to understand as long as we are in Papua New Guinea we are not safe and we are still in limbo. We need a safe, permanent place - settlement in a third country," he said.
"We have been kept to protect Australian borders for almost seven years, how long are they going to keep us for? It has to end. It's so long."
"It's dehumanising and a real crime. We are human beings like everyone else and we would like to live our lives peacefully and progress in our lives and contribute to the community and to the world."
A spokesman for the Royal Papua New Guinea Constabulary said police had not received any reports of asylum seekers being targeted in Port Moresby.
He said crime was a problem for everyone, saying "everyone is facing the same thing ... wrong place, wrong time, these things happen".
The Australian government's Smart Traveller website says there is a "persistent high level of crime" in the capital.
The spokesman added he believed the accommodation provided to asylum seekers there was "very good".
'People could become destitute'
The Refugee Action Coalition said "around 200" people have been transferred from Manus Island to Port Moresby.
The advocacy group, who is regularly in contact with asylum seekers in PNG, said the individuals are now staying at hotels with different degrees of support.
"It's just absurd … The government has tried to dress this up as somehow closing detention in Papua New Guinea when all that's happened is that detention has shifted from Manus to Port Moresby," spokesperson Ian Rintoul said.
Source: Refugee Action Coalition
"[And] it looks like they are fragmenting the groups, putting them in different hotels."
Mr Rintoul said Port Moresby "is a dangerous place and even more dangerous because the refugees are so visible".
He also voiced concerns about a "meagre" living allowance of 100 kina, approximately $50 a week.
"People could become destitute," he said.
Marcella Brassett, a campaign manager at the Asylum Seeker Resource Centre, echoed the criticism about the transfers.
"After holding people for six years on Manus Island and then suddenly moving them to a new location where they don't have an outcome for a permanent home is not the answer," she said.
"There is no future for them" in Papua New Guinea, she said, urging the government to take a resettlement offer from New Zealand.
Since 2013, New Zealand's government has publicly offered to settle at least 150 refugees being held on Manus Island or Nauru, but it has been rejected by successive Australian prime ministers.
"The New Zealand offer gives them a permanent home, where they can begin a new life with the freedom to move and the freedom to work and the freedom to settle."
Advocates have also voiced concern for the dozens of men whose asylum claims have been rejected that are being "held incommunicado" by PNG in the high-security Bomana immigration centre.
"It's a black hole ... [There is] no access to phones, no visiting arrangement, no contact with legal advocates," Mr Rintoul said.
"We know from the snippets that do come out they are in very desperate circumstances. All of them have lost a lot of weight."
Australian taxpayers paid $20 million for the construction of the Bomana centre, designed to house non-refugees ahead of deportation.
'A matter for PNG'
SBS News contacted the Department of Home Affairs about the transfers and future for those now in Port Moresby.
"The move from Manus to Port Moresby is a matter for Papua New Guinea Immigration and Citizenship Authority," a spokesperson for the department said.
Regarding Bomana, the spokesperson pointed to comments made by the commander of Operation Sovereign Borders Craig Furini at Senate Estimates last week.
"The Bomana Immigration Facility is operated solely by the government of PNG ... we have no visibility of what goes on inside," Mr Furini said.
Australia's involvement on Manus Island has been dramatically scaled back this year and PNG is now overseeing arrangements there.
In September, PNG announced it was ending a contract with Paladin, the firm running the camps on Manus Island that was awarded a $423 million contract in 2017 through a closed tender process.
Local authorities have said the transfers are a solution to a contentious years-long issue.
PNG immigration and border security minister Petrus Thomas told the Papua New Guinea Post Courier it was a "positive move for these men".
"Living in Port Moresby will enable them an opportunity to make their own personal decisions and move on with their lives … We wish these men the very best with their new lives."
In a separate Post Courier article, Mr Thomas said the transfers also benefit Papua New Guineans on Manus Island.
"With the end of regional processing, the Manusian community will be able to rebuild their identity separate to this issue," he said.
Mr Thomas said the East Lorengau Refugee Transit Centre will now be repurposed as a "dedicated education facility for the Manus community".
Additional reporting: Rosemary Bolger