Only one in four voters believe Dyson Heydon should continue as unions royal commissioner.
The Essential poll result on Tuesday came as the former High Court judge continued to consider a unions application for him to step down over perceived bias.
Mr Heydon was due to deliver his ruling on Tuesday but has given himself more time and suspended hearings while he does so.
The poll found 38 per cent of voters believe Mr Heydon should step aside in the wake of him accepting an invitation to speak at a Liberal Party event, which he subsequently withdrew from.
While 25 per cent of voters said he should stay, 37 per cent declined to express a view.
However, voters want the royal commission to continue should Mr Heydon decide to disqualify himself.
The Essential poll found 39 per cent believed it was a legitimate investigation, while 27 per cent said it was a political attack on Labor and the unions. Thirty-four per cent of voters declined to give a view.
Prime Minister Tony Abbott told reporters on Tuesday the inquiry should continue with or without Mr Heydon.
"This inquiry is not a political game, this inquiry is important for the future of the union movement and the Labor party," Mr Abbott said.
"So regardless of what the royal commissioner ultimately decides, the royal commission must and will go on."
Labor leader Bill Shorten said federal police were best placed to investigate corruption.
"Mr Abbott's royal commission is dangerously adrift and it's turning into quite the shambles, which unfortunately Labor predicted it might become," Mr Shorten told reporters.
A spokesman for Mr Heydon said in a statement on Tuesday: "Commissioner Heydon is taking the time required to consider his decision."
"A further update will be provided in due course." said.
Share

