Former Labor leader Simon Crean blamed "that perception argument" for why Mr Rudd's leadership was over before the caucus vote happened 10 days ago.
Ms Gillard indicated she was going to challenge on June 23; in the caucus vote the next day, Mr Rudd, who did not have much support, did not stand against her.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
Some observers argued that even if he had had the numbers to survive the challenge, his leadership would have appeared shaky and it would have meant the end for him.
"Once someone was being challenged - and (Mr Rudd) called the challenge on essentially, he said he was going to call the caucus together - even if he had the numbers quite frankly he couldn't win," Mr Crean, the education, employment and workplace relations minister, told the Nine Network.
"This is the same decision I had to make about my position some years ago."
Mr Crean was Labor leader from 2001 to 2003, surviving one leadership challenge before standing down after losing his party's support.
He said Mr Rudd's leadership hung on whether there was a challenger, which became clear on the night of June 23 in a meeting between Ms Gillard and Mr Rudd.
"It was only in the circumstances of their one and one conversation that the challenger emerged, and once that call was made, it was over for Kevin quite frankly, regardless of the
factional manoeuvreings."
Reflecting on his own experiences, he said: "Politics is a brutal game, it is a blood sport ... I've been through it."
Mr Crean, who was promoted in the ministerial reshuffle that followed Ms Gillard's takeover, rejected the common perception that Labor's factional leaders brought down Mr Rudd.
"I don't think it was the factions that brought him down."
