The groups say it highlights Australia's lack of transparency and accountability at the centres and it is time the country's new leader relaxed the Government's hardline policies.
The Federal Government invited the United Nations special rapporteur on the human rights of migrants, Francois Crepeau, for a visit before October 9.
Mr Crepeau was due to assess the situation of migrants and asylum seekers in the country, as well as Australia's offshore detention centres in Nauru and Papua New Guinea.
But he postponed his visit over concerns related to accessing the facilities and their staff.
Mr Crepeau says, under the Border Force Act 2015, detention centre workers who disclose what is deemed "protected information" can be sent to jail for up to two years.
In a statement, Mr Crepeau wrote:
"This threat of reprisals with persons who would want to cooperate with me on the occasion of this official visit is unacceptable."
He added he had asked the Federal Government for a written guarantee that no-one he met with would be sanctioned or intimidated.
"As the Australian government was not prepared to give the written assurances required ... it was not possible for me to carry out the visit in my capacity as a UN independent expert."
Mr Crepeau says his concern was it would discourage people from fully disclosing information.
Refugee Council of Australia spokesman Tim O'Connor says the Government has put detention-centre workers in a terrible situation.
He says Australia needs all the independent monitoring the United Nations and others can muster.
"It's extremely concerning that someone who does speak out about psychological, sexual or physical abuse of someone who is in the care of an Australian company funded by Australian taxpayers could be liable for two years' jail if they do so. Even speaking to a UN special rapporteur on the human rights of migrants could put those people in jeopardy of spending two years in jail."
Mr Crepeau says he is also extremely disappointed the Government did not grant him access to an offshore detention centre, despite months of efforts to gain its cooperation.
The humanitarian organisation Save the Children says the United Nations plays an invaluable role in watching over Australia's detention centres.
The group's spokesman, Tim Norton, says it raises questions over the fates of those in the centres.
"The UN can play a diplomatic role at the international level to come in, provide that oversight and liaise with the government about when and when they're not actually upholding human rights. To not allow that access to happen, there's no guarantee of what's going to happen to those children and those families in those facilities."
Deputy Labor leader Tanya Plibersek has told the ABC the Government must grant the appropriate protection to those who speak out and must let the United Nations do its job.
"We need to be careful with the personal details of refugees or asylum seekers in detention, but the idea that people couldn't confidentially inform the special rapporteur of what's going on is ridiculous, and I think particularly, given that the Australian government has said -- well, Julie Bishop has said -- that the Australian government wants to pursue a seat on the Human Rights Council in 2018, it's pretty contradictory."
Tim O'Connor, with the Refugee Council of Australia, says the council will be working with Australia's new prime minister, Malcolm Turnbull, to improve the situation for asylum seekers.
"We're certainly very hopeful, under the new leadership of the Government, there will be some compassion, there will be some humanity, and Australia will meet its commitments to ensure that we give safety and protection to vulnerable people who come to us seeking asylum."
Attorney-General George Brandis has rejected claims the Government does not cooperate with the United Nations on human-rights issues.
Rights groups and refugee advocates have claimed asylum seekers are subject to indefinite detention in inadequate conditions, with particular concern about the safety of children.
Share
