Labor says Special Minister of State Mal Brough is teetering on the edge over the Ashby-Slipper affair.
And Opposition Leader Bill Shorten says sooner or later, Prime Minister Malcolm Turnbull will have to make a call.
"Mal Brough's position is untenable," Mr Shorten told ABC TV on Sunday.
But Liberal Peter Dutton does not believe that a case has yet been made to sack Mr Brough from the frontbench.
"The most important thing is for the police to conduct their investigation and for that to continue without pressure," he told Sky News.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
"It will be resolved one way or another."
Labor is asking the Speaker for a parliamentary privileges committee inquiry into whether Mr Brough misled parliament when he denied asking James Ashby - a then staffer to speaker Peter Slipper - to procure copies of his boss's diary.
Federal police are investigating the Ashby-Brough matter, which carries a penalty of two years in jail.
