"It’s not about the public mood, it’s about what is the right decision for Australia’s national interest to ensure that the integrity of our border protection regime is maintained," Mr Morrison told reporters in Sydney on Monday morning.
Mr Morrison said making a last-minute ministerial intervention to allow them to stay would send a dangerous message to other asylum seekers.
"The worst possible thing we could do is... send a message of 'you know what if you come illegally to Australia, and the courts say you don’t have a claim and the government says you don’t have a claim, then the government just might make an exception because there’s been a public reaction'," he said.
"That's not how you run strong borders."
The prime minister also suggested the family could make another visa application to come to Australia after they return.
"I would hope they do," he said.
Barnaby Joyce backs family
Home Affairs Minister Peter Dutton earlier said he wouldn't take a "moral lecture" from supporters of a family who he accused of making false claims.
Mr Dutton defended the government's decision to deport Priya, her husband Nadesalingam and their Australian-born daughters Kopika, 4, and Tharunicaa, 2, in the face of criticism from Labor and the Greens.
Former Nationals leader Barnaby Joyce also spoke out in favour of allowing the family to stay.
"The kids ... they are born in Australia. They are Australians as far as the community is concerned," he told Seven's Sunrise.
"They are not their parents and I think we have to consider it in a different light. Time has moved on."
But Mr Dutton accused supporters of the family, who had made Biloela in Queensland their home, of peddling "false claims".
"We won’t take a moral lecture when the reality is we have a compassionate approach that is helping thousands each year, but where somebody has been told consistently all the way through to the High Court that they are not refugees, then those people have to return back to their country of origin," Mr Dutton wrote.
He told the Today show on Monday morning that the government had helped thousands of people that do not seek media attention.
"We deal with thousands of cases each year where we provide compassion in cases where kids might sick, where an elderly parent visiting Australia who may have had a stroke or other really difficult circumstances and we intervene in those cases and we should be very proud of that," he said.
But he said the Tamil family's case was "completely without merit in terms of their claim to be refugees".
"I understand the compassion shown by many, but we have to look at this case on its merit," he said.
Move to Christmas Island to avoid protesters
The family were transferred from Darwin to Christmas Island on the weekend, prompting complaints from their lawyers and advocates about difficulty accessing them.
Mr Dutton said the family had been moved to Christmas Island so they could be flown back to Sri Lanka without protesters putting Border Force officials in a "difficult position".
He said he did not want a repeat of the "volatility" at Melbourne airport on Friday night when two protesters were arrested for crossing a fence.
"These advocates lie about the way in which this family has been treated, that they've been forced, they've been pulled by the border force officials. It's complete nonsense," he said.
'The threat is real'
The Home Affairs Minister has also warned of the "real" threat of asylum seeker boats from Sri Lanka as it emerged a vessel was stopped by the Australian Border Force last month.
The boat from Sri Lanka was intercepted west of Christmas Island on 7 August, with 13 suspected asylum seekers flown back to the south Asian country on 18 August, The Courier-Mail reports.
Four asylum seeker boats carrying more than 90 people have now been intercepted off Australia since the May 18 federal election.
Home Affairs Minister Peter Dutton said the threat out of Sri Lanka was concerning.
"It is the reason Sri Lanka was the first country I visited after the election, to make sure we can keep these boats stopped. This threat is very real," he told the newspaper.
"Labor wants people to believe that the threat of new boat arrivals is not real. It is. We are dealing with it every day."
News of last month's asylum seeker boat comes as a Tamil family being held on Christmas Island are waiting to find out if a court will allow them to stay in Australia.
An interim injunction blocking the deportation of Priya, her husband Nadesalingam and their Australian-born children has been extended until Wednesday.
Additional reporting by AAP