He did not spare his old party in an interview on Monday, accusing the ALP of shifting its position on climate change as if it were in a yacht race.
Mr Latham said Senator Brown was the glue that bound the Greens together and he was a healer, a consensus politician and soft and gentle in his own personal style.
But there was a much more extreme faction in NSW, he said.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
"These tensions are coming to the surface and when Bob Brown retires, as eventually he will, I'd expect they will have real internal problems in holding their discipline and cohesion."
But Senator Brown rejected Mr Latham's criticism of the Greens.
"If there's one set of commentators who've always been wrong on the Greens, it's Labor commentators," he told ABC Radio.
"We're very robust, very healthy. The most vigorous party."
Last week, Prime Minister Julia Gillard said the Greens - with whom she has formed minority government - were a "party of protest" and did not share the values of "everyday Australians".
However, Mr Latham said Ms Gillard's Whitlam Oration was flimsy and shallow in failing to outline Labor ideology and face up to deep-rooted structural problems.
He said Labor was "chopping and changing, flipping and flopping" on climate change, showing it was not a party of fight and conviction.
"It's a party that is tacking on this issue like they are in a yachting race - oh, the wind's blowing a bit over towards the Greens, oh, we better tack that way," he said.
"Oh the wind's blowing back, the outer suburban people don't like it because of cost of living pressures, we better tack our boat over that way."
In an update of his 2005 'Latham Diaries', Mr Latham says the future of Australian politics lies with the Green movement not with the Labor movement.
