"It doesn't really matter whether Julia Gillard and I talk about this, what matters is that the government put the right policies in place," Mr Abbott told reporters in Sydney.
Liberal backbencher Mal Washer and Independent Tony Windsor have both called for a compromise on refugee policy.
Mr Abbott said he had not been approached by the government for talks to break the deadlock, but even if they did, the coalition would be sticking to the Howard government's Nauru policy.
"I'm all in favour of talking but in the end ... what matters here is the right policies, not talking endlessly about what they might be," he said.
The comments come as a search and rescue operation continues for up to 100 asylum seekers still unaccounted for after their boat capsized near Christmas Island on Thursday.
Three bodies have been pulled from the sea and 109 survivors were taken to Christmas Island on Friday.
Mr Abbott said he won't be criticising the government while the "tragedy" is still unfolding.
"I am doing my best to avoid (politicising the issue), not a word of criticism will escape my lips today," Mr Abbott said.
"We should be grieving for those who are lost," he said.
