The calls come as a new parliamentary inquiry is established into the treatment of asylum seekers in the offshore detention centres of Nauru and Manus Island.
Immigration Minister Peter Dutton is the father of three children, but potentially the legal guardian of many more.
As the immigration minister, he is legally responsible for all unaccompanied children seeking asylum in Australia.
Legal guardians are appointed to represent the best interests of a child but the Australian Council for Social Services, ACOSS, and the Australian Council for International Development, ACFID, say that cannot be done by the Minister.
Marc Purcell is the Executive Director of ACFID.
"A guardian should be appointed because patently the minister for immigration has a conflict of interest. He is charged with both caring for children (and) also detaining them in an environment where they're being harmed."
Both groups are also calling for the immediate release of all children from detention.
Their calls echo the recommendations of numerous reports, including last month's Senate Inquiry.
Previous reports such as the Moss Review and the Forgotten Children Report also highlighted allegations of assaults and and sexual abuse in immigration detention centres, including against children.
ACOSS and ACFID want those allegations examined by the existing Royal Commission into institutional responses to child sexual abuse.
Dr Cassandra Goldie, from ACOSS, says claims of child abuse in detention centres should be subject to the same level of scutiny as those that occur in any Australian institution.
"So that we can, as a community, ensure that all the investifations have been made and recommendations are developed to prevent any further risk of children being abused in these kinds of facilities."
There are currently 97 children in immigration detention in Australia and 92 on Nauru.
Mr Purcell says that's too many.
"Any Australian, any politician can look into the eyes of their own children or grandchildren and see that potential of children looking back at them, and know that we are damaging that potential of children currently by holding them in detention here in Australia and also on Nauru."
Dr Goldie says it's important children are taken out of a setting which she says is causing them harm.
"Children need to be moved out of detention facilities and away from Nauru, and placed into the community, in settings that are suitable for children. And to make sure that their current mental health and other needs are urgently addressed. I think the voice of ACOSS the voice of the Australian Council of International Development, and right across the community must be heard by the government - that there can be no policy that justifies putting children at risk."
The conditions in offshore detention centres on Manus Island and Nauru will be further investigated by a parliamentary inquiry.
Earlier this week, thousands of doctors at Melbourne's Royal Children's Hospital refused to discharge asylum-seeker children back into the care of immigration detention centre operators.
