The manager of opposition business Tony Burke moved the motion of no confidence in Speaker Bronwyn Bishop during question time, saying she is too partisan.
The move followed Ms Bishop's upping the ante in her ongoing battle with Labor, banning shadow attorney-general Mark Dreyfus from parliament for 24 hours.
It is the first such motion since 1949.
The motion was rejected 83 votes to 51, but not before heated debate between members of both the Labor and Liberal parties.
The minister for opposition business said the speaker had shown partiality in favour of government members and acted as an "instrument of the Liberal Party" who "constantly fails to interpret standing orders" and showed "gross incompetence".
He said the speaker had so far thrown 98 people out of Parliament, all of them Labor Party members.
"This resolution today is not one that people rush to move. This is a resolution - and every time, whether it is a suspension of standing orders or whether leave is granted, on every occasion that a resolution of this nature is moved, it is carried forever in practice," Mr Burke said.
"For the very simple reason that while Opposition members when they get to this point don't expect to win the vote, they do expect to have a situation where everyone in Australia knows bias when they see it.
"You're effective as a warrior for the Liberal party, but that is not the job you chose to take on. In the Speaker's chair you have continued to act as though enjoying the victory for your own side is your job.
"The parliament deserves more than that and the parliament cannot have confidence in a Speaker who refuses to be impartial."
Christopher Pyne described the motion as a stunt and said the opposition needed to toughen up.
"I am no sook. I have been manager of opposition business for five years. I was manager of opposition business for three years in a hung parliament. I hold the record for being ejected from this place by Speakers in the parliament," he said.
"I never complained. I never complained. I didn't stand up like a great big sook like the manager of opposition business did today and say like one of my four children that I have had my toy taken away from me.
"I know opposition is tough. It is not challenging, it is not satisfying. You don't get to make any decisions.
"Paul Keating put it very well in a debate on the matter of public importance in response to the then member for Flinders when he said: 'Honourable members opposite have three more years of their lives trotting around in opposition, three more years in the corridors at night wandering in and out of each others' offices having cold cups of tea at 11 o'clock.'
"You fall silent because you know it is true. The sadness is for the opposition, you lost the election. You have three years, hopefully more, in opposition and you just have to get used to it.
"When you're in opposition, you do get thrown out of parliament more often than members of the government.
"When I was in government, I was thrown out of the chamber as was the leader of the House. You have to put up with it. That is the way it is."
How it happened: SBS Chief Political Correspondent Karen Middleton reports from Parliament House
Share
