A group of 25 asylum seekers refused refugee status has staged a protest at the Christmas Island detention facility.
And Immigration Minister Chris Evans has forecast more asylum seekers are likely to be refused protection.
"When people are refused they are harder to manage and so we are going to have challenges there," he told Fairfax Radio on Thursday, adding there was "a bit of a protest" at the facility on Wednesday night.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
"And there's more to come."
Senator Evans said he understood the rejections did not go down well with people.
Many asylum seekers originating from Afghanistan were from the much-persecuted minority Hazara ethnic group.
"What we have found is that the situation for Hazaras in Afghanistan is improving," Senator Evans said.
That was why the government had introduced a six-month pause in the processing of new claims by Afghan asylum seekers and why earlier applications were being refused.
New applications from Sri Lankan arrivals have been put on hold for three months.
Senator Evans said there had been few arrivals from Sri Lanka in recent months although the Malaysian government had intercepted one group thought to be heading to Australia.
"So I don't want to pretend the problem has been solved," he said.
Australia had returned people to Sri Lanka, including Tamils.
"The numbers being refused is increasing as the situation in Sri Lanka improves."
Senator Evans acknowledged the number of asylum seekers arriving by boat - more than 2000 so far this year - was causing problems for the government in the leadup to the election.
"It's hurting us, no question about that, because people in Australia are rightly concerned about people arriving unauthorised," he said.
Many recent arrivals who gained refugee status after being assessed offshore, especially those from Africa, were concerned that unauthorised arrivals were limiting their prospect of bringing family members to Australia.
"There is no doubt it increases tensions," Senator Evans said.
