The group said going on the planned December 6 trip without being able to speak privately to inmates "would have created a disastrous precedent."
"Since the Americans have not accepted the minimum requirements for such a visit, we must cancel," Manfred Nowak, the UN envoy in charge of investigating torture allegations around the world, said.
"It means there will be no visit this year for the report we are preparing, but naturally we are totally prepared to go to Guantanamo in the future if the Americans assure us of their full cooperation."
US State Department spokesman Adam Ereli said the United States had been receptive to UN requests, but added, "If they aren't satisfied, well, sorry."
"Frankly, we believe we've been very forthcoming in response to their requests. We have offered them the same access to this facility as we offer elected representatives of the American people," he said.
"Members of Congress have come to visit Guantanamo. And the (UN) special rapporteur is receiving the same treatment as members of Congress.
The December 6 date had been set after more than three years of discussions between US and UN officials amid claims of human rights abuses at the camp.
"Under the circumstances, we will not be traveling to Guantanamo," the five UN experts, including Mr Nowak, said in a statement in Geneva.
"Doing so would undermine the principles of UN human rights fact-finding missions," it added.
The US on Tuesday refused the UN human rights monitors' demand for an unconditional inspection of the Guantanamo camp which they had said was standard practice for a "credible, objective and fair assessment."
The US State Department said Washington was open to inspectors but slammed the way the UN experts had piled up public pressure for access.
The US provided regular access to the International Committee of the Red Cross (ICRC) and had consulted with governments on the cases of their nationals detained there, the State Department said.
The UN experts maintain that the ICRC's monitoring is very different.
The ICRC as a matter of policy does not make its findings on humanitarian conditions public in order to preserve access to prisons that may otherwise be closed to them.
The UN experts in contrast are mandated to investigate allegations of human rights breaches and report publicly to the UN General Assembly and the UN's watchdog, the Human Rights Commission.
The US government has been sharply criticised for conditions at Guantanamo, where 520 detainees are held without trial and where some have gone on hunger strikes.
Most of those held there were captured after a US-led offensive toppled the Taliban regime in Afghanistan in late 2001.
The US has declared the detainees to be illegal enemy combatants not protected by the Geneva Conventions.
Even without a visit, the UN monitors still intend to write a report on conditions at the prison based on eyewitness accounts from released detainees, meetings with lawyers and information from human rights groups.
Citizenship rejected
The British government negotiated the release of all nine of its citizens held at Guantanamo Bay.
Two Australians were detained there, one has been released without charge, the other, David Hicks, is awaiting trial.
Britain this week rejected a citizenship application from Mr Hicks, who is alleged to have fought alongside the Taliban in Afghanistan against US-led forces.
One of his lawyers said the application was denied because Mr Hicks had allegedly performed "an act prejudicial to the interests of the United Kingdom in attending training camps in Afghanistan and Pakistan".
The decision is set to be appealed before a judicial review panel on Wednesday.
David Hicks's trial by military commission was scheduled to begin on Friday but a US federal judge ordered the suspension of proceedings this week ahead of an anticipated ruling by the US Supreme Court on the legality of the military tribunals.