Addressing the nation on television shortly afterwards, Abiy said the attack was an “attempt by forces who do not want to see Ethiopia united.”
Abiy had promised in his rally speech to bring more transparency to government and reconciliation to a nation torn by years of protests. When speaking on television, he was still wearing a green T-shirt handed to him by rally organizers.
He said several people had been killed, but his chief of staff Fitsum Arega later said on Twitter that there were no deaths.
"As of now, based on reports from police and hospitals, 83 people are injured. Of the 83 injured, six are in critical condition. No death so far has been reported," Arega said.
"Some whose heart is filled with hate attempted a grenade attack," he added, vowing that the perpetrators will be brought to justice.
"The people who did this are anti-peace forces. You need to stop doing this. You weren't successful in the past and you won't be successful in the future," he said.
The AFP correspondent said the blast happened stage right from Abiy and sent up a small puff of black smoke.
More than 100 people then stormed the stage, hurling various objects at police, shouting: "Down, down Woyane," and "Woyane thief", using a pejorative term for the government, the AFP correspondent said.
Police later used tear gas to clear the area, while an AFP photographer saw two men and two women taken into custody on suspicion of being involved with the blast.
The rally in the central Meskel Square was Abiy's first public speech in the capital since he took office in April, although he has made several in provincial areas.
In the past three months, Abiy has made major changes in Ethiopia including shaking up the security services, releasing jailed dissidents, moving to liberalise the economy and to resolve a two-decade conflict with arch-enemy Eritrea.
"Ethiopia will be on top again, and the foundations will be love, unity and inclusivity," he declared, drressed in a green T-shirt and a hat.
Opposition ends 'armed resistance'
Abiy succeeded Hailemariam Desalegn, who resigned in February amid a wave of anti-government protests led by the country's two largest ethnicities, that started in late 2015 and left hundreds of people dead.
While it remains unclear how deep Abiy's support runs within the ruling Ethiopian People's Revolutionary Democratic Front (EPRDF), his actions thus far represent dramatic shifts in the power balance within Africa's second-most populous country.
Political rallies of Saturday's scale are rare in Ethiopia, where the EPRDF controls all seats in parliament and opposition parties complain of harassment.
But at the rally people openly displayed flags of banned groups such as the Oromo Liberation Front, an act that would usually result in arrest.
Abiy's reforms have spurred some anti-government groups to seek rapprochement.
After the release of top official Andargachew Tsige in May, anti-government group Ginbot 7 announced on Friday it would cease armed attacks in the country, citing Abiy's reform agenda.