Tom Calma, the former Aboriginal social justice commissioner, called on politicians to stop the infighting and get on with the job of bringing the Indigenous community out of disadvantage.
"We should now move away from using Aboriginal affairs as the political football for ministers to fight their battles," Professor Calma told ABC Radio.
Mr Scullion, who has been heavily criticised for his handling of the NT's Don Dale juvenile detention scandal, has stressed he won't make way for the former Liberal Party leader.
Mr Calma called on Mr Scullion to resign at the time of the Don Dale saga, and on Wednesday said neither he nor Mr Abbott are particularly well equipped for the role.
"They've both got a track record that's not appealing," he says.
"If they work together, one from the backbench (and) the other in cabinet, they could do a lot more to advance Indigenous affairs."
Mr Abbott wants the role so he can re-enter cabinet and heal the rift with Prime Minister Malcolm Turnbull, who deposed him last year, according to his close friend and columnist Catherine McGregor.
