Mr Annan says it is heartbreaking the lives of Iraqis are dominated by “the constant threat of sectarian violence and civil strife."
He made the comments at a special meeting on Iraq held at the UN headquarters in New York which was attended by the Iraqi President Jalal Talabani and US Secretary of State Condoleezza Rice.
Mr Annan said the country and its leaders are now at "a crossroads" and action must be taken in Baghdad and internationally to "bring Iraq back from the brink".
The UN chief has warned there is a grave danger that the Iraqi state will break down, possibly in the midst of full-scale civil war.
Worst day of violence
Suicide bombers claimed the lives of more than half of the 62 people killed in the worst violence in Iraq in weeks.
One bomber targeted Iraqis waiting for their butane gas ration cards in the northern city of Tal Afar, killing 21 people.
Police say the bomber detonated an explosives belt he was wearing as he approached a crowd waiting for the cards in central Tal Afar.
A second bomber killed 13 people when he struck a police station in the Sunni insurgent stronghold of Al-Hurriyah.
In other attacks, 28 people were killed including four members of a Shi'ite family shot dead by gunmen as they tried to flee their homes north of Baghdad.
The Iraqi government nevertheless has given an optimistic assessment of the impact of a massive security operation in Baghdad.
Tens of thousands of Iraqi and US security personnel have been deployed on the streets of Baghdad since June.
The deployment is part of a bid to stem sectarian violence between the ousted Sunni Arab elite and the newly empowered Shi'ite majority.
In the past week, however, the violence has spiked with dozens of bodies turning up everyday, including 14 found in the heart of the Baghdad today.
”Peace still within reach”
Mr Annan believes the promise of peace and prosperity will remain in reach if Iraqi leaders address the needs of ordinary Iraqis.
He said Iraq had made "important progress" in the past two years with two national elections and a constitutional referendum.
The UN chief called on Iraq's leaders to make stronger efforts to "defuse rising sectarian tensions."
He backed an Arab League proposal to hold a national conference in Baghdad to discuss issues such as federalism and the sharing of revenues.
"Many of Iraq's neighbours have legitimate concerns about instability inside that country. Iraq must remain sensitive to these apprehensions. The neighbours, for their part, must also be responsive to Iraqi security needs," he said.