But former prime minister Malcolm Turnbull says allegations made against Mr Heydon by six women are "shocking and appalling" while Prime Minister Scott Morrison has branded the claims "very disturbing".
An independent inquiry initiated by the High Court, revealed this week by the Herald, found Mr Heydon sexually harassed six judges' associates.
High Court Chief Justice Susan Kiefel said this week she was ashamed the alleged harassment could have happened at the High Court.
"The findings are of extreme concern to me, my fellow justices, our chief executive and the staff of the court."
Speaking through his lawyers, Mr Heydon categorically denied the women's claims of "predatory behaviour".
"Our client says that if any conduct of his has caused offence, that result was inadvertent and unintended and he apologises for any offence caused."
Mr Turnbull says he was absolutely shocked at the news.
"There is no way you can describe this other than shocking and appalling," he told the ABC.
"This is about power. This is about men abusing their power over women and we must have zero tolerance for it. We have to have respectful workplaces."
Three of the women are pursuing claims for compensation.
Their lawyer, Josh Bornstein from Maurice Blackburn, said the firm would first try to negotiate an outcome with Mr Heydon's lawyers.
Failing that, they will pursue the matter with the Australian Human Rights Commission, and then through the Federal Court if it's allowed.
Mr Morrison, responding to calls Mr Heydon should be stripped of his Companion of the Order of Australia honour, said it would be reviewed if the allegations are upheld.
Mr Heydon sat on the High Court bench from 2003 to 2013.
Then, in 2014, he headed up a royal commission into union corruption set up by the Tony Abbott-led Liberal government.
The independent report was prepared for the High Court by former inspector-general of intelligence and security Vivienne Thom.