Scott Morrison has refused to apologise.
Mr Robert said the former postal boss should have waited for the outcome of an independent review, which later cleared her of any wrongdoing, before resigning from the role.
"We should always wait for independent investigations. We shouldn't jump the gun. We should always wait to see what they say and we should act on them," he told the ABC on Friday.
Mr Robert is no stranger to scandals over luxury watches.
Several years ago, he was forced to return an expensive Rolex timepiece given to him by a Chinese billionaire.
Mr Robert denied there were double standards at play, despite keeping his job while Ms Holgate did not.
The minister claimed the distinction lay in him believing the Rolex was not real.
"We thought they were glittery fakes," he said.
"We got them valued, we found they were real, we then returned them voluntarily."
Mr Robert was also forced to repay $38,000 to the government for excessive home internet bills.
The amount he charged taxpayers was almost double the cost of the Cartier watches bought by Australia Post.
The minister argued he was not to blame because the finance department set up the connection.
"I was the one that voluntarily repaid it based on something that was set up for me, so again, the circumstances are fundamentally different," Mr Robert said.