"My decision is to be part of the ongoing reform programmes," Hailemariam said, according to state-affiliated Fana Broadcasting Corporate.
"I believe my party and government will make history again by conducting (a) peaceful power transition," he added.
Hailemariam will remain in power until parliament and the full council of the ruling coalition party, the Ethiopian People's Revolutionary Democratic Front (EPRDF), confirm his resignation. It remains unclear who will take over after that.
The streets of the capital Addis Ababa were calm following the surprise announcement.
Ushered into office after the death of former prime minister Meles Zenawi, a one-time Marxist who had led the rebellion that overthrew the Communist Derg regime, Hailemariam transformed from a relatively little-known politician to a technocrat and influential leader.
While Ethiopia is one of Africa's poorest countries, Hailemariam continued a streak of rapid economic growth.
Unlike many in the ruling elite, Hailemariam was not part of the rebel movement which toppled the Derg. Instead he was studying civil engineering in Addis Ababa, was completing his master's degree at Finland's Tampere University when the dictator fell.
Political analyst Hallelujah Lulie said the seeds of Hailemariam's departure were sowed in 2015, when anti-government protests erupted that led to hundreds of deaths and prompted the government to impose a 10-month state of emergency in October 2016.
While the decree halted the unrest, protests still erupted occasionally and the upheaval laid bear divisions within the four ethnically based parties that under the EPRDF umbrella have ruled Ethiopia since 1991.
"There's a general consensus that the way he handled the crises was not all that good," Hallelujah said.
"The divisions within the EPRDF did not heal, and we can’t tell whether they’re going to be healed or not," he added.
In a bid to ease tensions, the government last month began issuing a string of pardons and prisoner releases, after Hailemariam said jailed "politicians" would be released "to improve the national consensus and widen the democratic platform".
Earlier this week, men wielding sticks and stones blocked roads around the capital and the Oromia region, which has been at the centre of the protest movement, in a two-day strike to push for more prisoner releases.
The developments have caused top elites in the military and intelligence services to lose confidence in Hailemariam, Hallelujah said, while the regional governments at the centre of the protests, Oromia and Amhara, have increasingly shrugged off federal government control.
"He is unable to control the party and he is unable to control the government, and I think it is the right decision that he resigned," Hallelujah said.
In a country long dominated by its major ethnic groups -- most recently the Tigray, the ethnic group to which Meles belonged -- Hailemariam notably comes from the minority Wolayta people in the Southern Nations, Nationalities and Peoples Region.
He was also the first Protestant to lead Ethiopia, where the majority of Christians follow Orthodox traditions.