A spokesman for Mr Annan said he has decided it would be inappropriate to include Iran in his upcoming tour through Middle Eastern countries which begins on the weekend.
"The secretary general and the Iranian government have mutually agreed that this is not an appropriate time for him to travel to Iran," said UN spokesman Stephane Dujarric.
"In light of the ongoing controversy, it would have been difficult to advance the agenda that he had wanted to discuss with the Iranian leadership."
He will now go to Egypt, Saudi Arabia, Tunisia and Pakistan, the spokesman said in a statement.
Mr Annan is travelling through Paris, where he is scheduled to meet with President Jacques Chirac on Monday.
While the statement did not specify what the controversy was, Iran has received numerous complaints after President Mahmoud Ahmadinjad last month called for Israel to be "wiped off the map" -- comments that reinforced worries about Iran's nuclear ambitions.
His remark was the first time in years that a top Iranian official has openly called for Israel's destruction.
He made the comments in a conference titled A World Without Zionism on October 26.
The UN Security Council issued a statement condemning the comments and Mr Annan said he was dismayed by the remarks, in a rare rebuke.
Israeli foreign ministry spokesman Mark Regev said his government "welcomes the international outrage that has been expressed to the obscene statements by the Iranian president and is heartened by the statements of world leaders who have condemned the remarks", the Associated Press news agency reported.
Israeli Prime Minister Ariel Sharon has called for Tehran's expulsion from the United Nations.
Mr Annan had been due to visit Iran from 11 to 13 November, in a trip that aims to focus on the Middle East, including regional stability, the Middle East peace process, Iraq, and the follow-up to September's UN World Summit.
Iran faces the risk of referral to the Security Council after the International Atomic Energy Agency in September found Iran in "non-compliance" with the nuclear Non-Proliferation Treaty.
Diplomats say Tehran has appeared more cooperative with IAEA investigators in recent days, seeking to avoid referral to the Security Council and possible trade sanctions.
Expressing dismay over the Iranian president's remarks, European Union foreign ministers were due to meet next week to review their policy towards Tehran.
Member states of the IAEA are due to discuss Iran's nuclear program at its headquarters in Vienna on November 24.