According to media reports two boats -- one of which was apparently carrying 150 people including Tamils -- were intercepted by Australian authorities off Christmas Island.
It's a development the federal government has refused to confirm.
And the Sri Lankan military says it's unaware of any arrangements with Australia to return asylum seekers.
"We are not aware of any arrangements of the Australian Navy handing over refugees, to Sri Lankan Navy," a spokesman for the Sri Lankan military told ABC Radio late on Tuesday.
'It is shocking to us'
The Australian Tamil Congress says one of its members was in contact with the passengers until late last week, when communication went silent.
Spokesman Dr Raga Ragavan says the organisation would be shocked to learn they'd been sent back.
"If they have been sent back and given over to the Sri Lankan navy, we can't believe that our government, the Australian government, supposed to be a humane government which is a signatory to the UN convention on refugees, are doing this," he told SBS.
"We can't believe it and it is shocking to us."
Refugee advocates say they haven't heard from the asylum seekers since Saturday and they believe Australia's coordinating with the Sri Lankan navy to tow them back.
One of the boats is reportedly carrying 153 Tamil asylum seekers, including about 30 children - many of them sick - while the other has 50 people aboard.
Christmas Island councillor Gordon Thomson has called on the federal government to confirm the accuracy of reports asylum seekers have been transferred from Australian custody.
Australians will be 'horrified': Greens
The Australian Greens say they're disturbed to hear reports the government's working with Sri Lanka to return Tamil asylum seekers.
"They have a shocking reputation for human rights," Greens leader Christine Milne told a press conference this morning.
"It is now up to Prime Minister Abbott. Prime Minister tell Australians: are you going to send 153 people back to the people who have persecuted them? Is that what this nation has become under your leadership? Because I think the overwhelming majority of Australians will be horrified by this.
"Not only is it shocking and cruel for the people who have been persecuted and are being treated like this, but it is absolutely in breach of our obligations under the refugee convention."
Abbott, Morrison refuse to comment
The government has repeatedly refused to confirm reports two boats, containing more than 200 people, have been intercepted by Australian authorities off Christmas Island.
Earlier on Tuesday, Prime Minister Tony Abbott refused to say whether the boat was being taken back to Sri Lanka.
"I'm not going to comment on the operational detail of what happens on the water," he told ABC Radio.
Immigration Minister Scott Morrison has also refused to comment, saying that public curiosity was not the same as the public interest.
"In accordance with the policy established by the Operation Sovereign Borders Joint Agency Task Force commander, the Government does not comment on speculation or reporting regarding on-water operations," a spokesman for Mr Morrison said.
"The Government will continue to act in accordance with our international obligations, including applicable international conventions and to protect the safety of life at sea."