Julia Gillard is believed to have the numbers going into this morning's Labor caucus leadership ballot.
But Senator Brown says it'd be a worry if Kevin Rudd's right that replacing him with Ms Gillard could see a shift to the right on asylum seeker policy.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
Family First senator Steve Fielding says Mr Rudd's leadership style has brought him undone.
He says Ms Gillard's able to consult something Mr Rudd hasn't been able to do.
Independent senator Nick Xenophon says Mr Rudd could have done more to cut through on policy.
Downer weighs in
Former Howard foreign minister Alexander Downer says Prime Minister Kevin Rudd - who is facing a leadership challenge - has never been popular with MPs.
Speaking of Mr Rudd's colleagues in the Labor party, Mr Downer said, "I never thought they liked him."
"Kevin Rudd has always been a very unpopular person in the parliament," he told Fairfax Radio Network from Cyprus.
"Popularity is not about whether you're are Liberal or Labor, you don't like people just because they are Liberal or Labor.
"He was just not a person that people liked, because he was so self-centred and so self-interested."
But the public had not caught onto that until recently, Mr Downer said.
"Then suddenly, particularly because of the dumping of his climate change policy, the public thought, hang on, this guy's not - as Kevin Rudd would say - fair dinkum, which he's not.
"And they drew the right conclusions about him, and it's the end of him of course.
"It's amazed me he's gone as far as he has, frankly."
Mr Downer said Deputy Prime Minister Julia Gillard, who is challenging Mr Rudd for the Labor Party leadership, was different.
"She's not an unpopular person in the way that Kevin Rudd was or is."
