Opposition Leader Bill Shorten believes Bronwyn Bishop's position as Speaker is no longer tenable and the prime minister should stand her down.
His demands came as Ms Bishop refused to say sorry for slugging taxpayers with a $5000 helicopter ride from Melbourne to Geelong to attend a Liberal fundraiser, saying paying the money back was an apology in itself.
Quizzed by reporters after an Anzac Centenary event in Sydney on Sunday, Ms Bishop also said she felt no particular pressure to resign.
But she was annoyed when asked to explain the difference between her case and that of Peter Slipper, who stood aside as Speaker during the Gillard government as an independent MP when cab chargers of some $900 were investigated.
"Totally a different matter all together and you know it," she retorted.
The helicopter trip and other expense claims are now under investigation by the Department of Finance.
Mr Shorten said this has now become a test for Tony Abbott's leadership who has so far stood by his hand-picked Speaker.
"Tony Abbott has to ask Mrs Bishop to stand aside," he told reporters in Melbourne. "He can't dismiss this as ... village gossip. It's this sort of arrogance which turns Australians off politics."
Ms Bishop wasn't getting much support from former Liberal MPs either.
John Hewson, who was Liberal leader when Ms Bishop was a senator two decades ago, said she set a pretty tough standard in those days for accountability and transparency, but is now "ducking and weaving" now that the same standards are being applied to her.
"It's amazing how history bites you in the bum," he told Sky News.
Former treasurer Peter Costello said she should have never taken the flight.
"She should also explain why this is parliamentary business," he told the Network Ten.
Treasurer Joe Hockey, who was the first minister to suggest the helicopter ride "failed the sniff test", said the Speaker had recognised she made an error of judgment and was co-operating with the Department of Finance.
"This has sucked up too much oxygen for the nation over the last few days," he told Network Ten. "We need to get on with the issues that matter to people."
Government frontbencher Greg Hunt agreed with the Speaker that her refund of the flight was as good as an apology but conceded he would apologise for any error he made.
Greens Senator Lee Rhiannon said the debacle was a "weeping sore" for the Abbott government and should be investigated by the Australian Federal Police.
If Ms Bishop signed for the $5000 declaring it was a legitimate expense, then that is fraud.
"If the AFP investigation confirms what appears to be evident, then the Speaker should be prosecuted," she said. "If convicted Mrs Bishop should resign from parliament."
