At 9.2 million people, the number of refugees was at its lowest level for 25 years, but this was tempered by a swell in the number of internally displaced people, refugees within their home country, to 25 million.
There was widespread confusion over migrants and refugees and the world was falling short of meeting internally displaced people's needs, high commissioner Antonio Guterres said.
"One of the main problems we face in order to safeguard asylum seekers is a growing intolerance, happening everywhere," said the former Portuguese prime minister.
"We are witnessing the rebirth of irrationality," he said, citing xenophobia.
"This is creating a difficult environment in which the foreigner is sometimes hated and feared."
Book launch
Mr Guterres was launching the Office of the United Nations High Commissioner for Refugees's (UNHCR) new assessment, "The State of the World's Refugees: Human Displacement in the New Millennium".
The book, which tackles how the changing dynamics of displacement over the past five years have affected the international asylum regime, said the refugee system had reached a critical juncture.
The drop in refugee numbers was mostly due to the end of several conflicts and a few large-scale repatriations, notably four million people returning to Afghanistan, and others to Angola and Sierra Leone.
However, the rise in numbers of internally displaced people represented the "biggest failure in terms of humanitarian action", Mr Guterres said.
"They are under the protection of their own government and their own government is in many circumstances part of the problem, not part of the solution," he warned.
However, he added that he would be against altering the 1951 UN convention on refugees to tackle the growing concern, because it would be likely to change for the worse.
’Asylum fatigue’
There were about 175 million international migrants in 2005, including refugees and some 838,000 asylum seekers.
But the distinction between illegal immigrants on the one hand and refugees and asylum seekers on the other had been blurred, the high commissioner said, with "asylum fatigue" setting in some countries.
He urged states to reconcile their legitimate concerns, such as migration controls, with their legal and humanitarian obligations towards uprooted people.
The book said that the provision of international protection and the application of international human rights and humanitarian principles were being "increasingly challenged by political, social and economic realities".
Mr Guterres also voiced his concern that international aid workers and forces protecting refugees were increasingly finding themselves in the firing line as they carried out their duties in conflict zones.
He cited the 200,000 Darfur refugees in Chad, their protectors and aid workers there as being particularly at risk.