Mr Abbott took the indigenous portfolio into his own department after winning the 2013 election and delivered on a pledge to spend a week each year governing from a remote community.
However, his term was cut short in September 2015 when Malcolm Turnbull ousted him.
Mr Pearson told the National Press Club on Wednesday Mr Abbott had been his "closest friend" in political circles and one of the few people on his mobile phone speed dial.
"It was cut short," Mr Pearson said of his friend's tenure.
"I think there's been few people more genuinely signed up to our cause than him. I regret his passing."
Mr Pearson said Mr Abbott had struggled over two years to come to terms with the size of the challenge in improving living standards for Aboriginal and Torres Strait Islander people.
But the former prime minister got the ball rolling on recognising indigenous people in the constitution.
"It was essential that a conservative start the kick-off ... Nixon had to go to China and a conservative had to kick this ball along the road."
It was now important that Mr Turnbull used his experience to facilitate the referendum.
Mr Pearson said he was disappointed the Abbott government had "inadequately" responded to a reform proposal put to it a year ago.
Mr Abbott has declared he will recontest his Sydney seat of Warringah at the next election.
Mr Pearson said he believed Mr Turnbull had received advice on how he might continue the "tradition" of working out of a remote indigenous community once a year.
1788 ‘one of three defining moments’ of Australian history
In his address, Mr Pearson also raised the controversial statement made in 2014 by then prime minister Tony Abbott that the arrival of the First Fleet on 26 January 1788 was Australia's most defining moment.
Mr Pearson said Mr Abbott failed to mention two more.
These, he said, were when more than 50,000 years ago, the first Australians crossed the Torres Strait land bridge to this continent, and the abolition of the white Australia policy between 1973 and 1975.
“It is these dates that speak to the three parts of Australia, our Indigenous heritage, the fact of our British colonisation and the removal of discrimination against migrants,” Mr Pearson said.
He adds Australia Day, January 26, would not be fuelled with so much “offense and hurt” if the day “spoke to these three parts”.
“It can't just be about what was destroyed. It must also be about what we have built. The meaning we invest in Australia can't just be about what was destroyed, but what was created.”
‘Wrong turn’ veering off political path
“It's my greatest regret,” he said when asked about the decision to decline a career in politics. “Fifteen years ago I began our reform journey in Cape York, 15 years ago, I was 35 years old. And I made the wrong turn.
“I made the wrong turn because I think I've hit the limit of how much influence you can have barking from the outside.”
Mr Pearson instead chose to lead Cape York Partnerships - the NGO which is advising government about the best measures to improve outcomes for Indigenous people.
It has proposed the government adopt an ‘Indigenous Empowerment’ policy, which aims to give Indigenous communities increased voice in decision making around Indigenous affairs through partnerships between communities and governments at local, regional, state and national levels.
“We've reached the kind of dead end of Indigenous affairs presided over by a Minister and a department,” Mr Pearson said.
“I'm not saying that the people involved are insincere. It is just that the system by which they attempt to deal with our communities is not one that works.
“It can't discern excrement from clay. It just cannot.”
Share
