"Pacific island leaders cannot ignore this issue any longer and need to ensure that it is at the very top of the forum's agenda," Amnesty's Pacific researcher Roshika Deo said on Thursday.
Source: RURAL AUSTRALIANS FOR REFUGEES
"This is a desperate situation that requires urgent action. Regional leaders must show that they will not stand by while the Australian government's abusive policies continue to risk more lives."
The rights groups said asylum-seekers on Nauru and PNG's Manus Island were subjected to "cruel and degrading treatment" that must stop.
"(There are) widespread reports of violence against refugees in Papua New Guinea and violence and sexual harassment of women and children on Nauru," the letter said.
There are more than 200 people in the Nauru facility, according to the Refugee Council of Australia, including dozens of children.
Rights advocates say they are suffering mental health issues under the strain of indefinite detention, with reports of despondent children harming themselves.
However, the Canberra-bankrolled facility has been an economic lifeline for Nauru, which has an area of only 21 square kilometres and has depleted its only natural resource, phosphate.
The Nauru government has imposed strict conditions on media covering the PIF summit, threatening to revoke journalists' visas if they capture images of the camps or asylum-seekers.
It has also limited the number of reporters attending and barred the ABC, after taking exception to its coverage.