“There is no torture, there is no abuse,” Mr Rumsfeld told an audience at the Council on Foreign Relations in New York.
A day earlier, UN chief Kofi Annan called for the closure of the Guantanamo Bay detention centre “as soon as possible”, saying that people could not be held indefinitely without charge or a chance to defend themselves.
“I know Kofi Annan and there a lot of things that you can agree with him on, but he’s just flat wrong,” Mr Rumsfeld said.
“We shouldn’t close Guantanamo,” he said.
“We have several hundred terrorists, bad people, people if they went back out on the field would try to kill Americans.”
The defence secretary went on to say that 15 of those detainees set free had returned to the battlefield.
"The idea that you could just open the gates and say, 'Gee fellas, you're all just wonderful,' is just not realistic," Mr Rumsfeld argued.
Mr Rumsfeld also charged Mr Annan of not knowing what conditions are actually like inside the facility.
“He hasn’t been to Guantanamo Bay,” Mr Rumsfeld said.
Neither have the five authors of the UN report.
A scheduled visit to the detention centre in December was cancelled after the group failed to get approval to speak freely with the prisoners.
Their research is based on the US government’s response to a questionnaire and interviews with released detainees and lawyers working for the freedom of those still in detention.
But the report has stirred up a wave of opposition to the US detention centre.
The European Parliament has passed a resolution calling for fair and immediate trial of prisoners.
Even key US ally, Britain has voiced its concerns, with British Prime Minister Tony Blair calling Guantanamo "an anomaly" that "sooner or later has to be dealt with."
The country's second-most senior Anglican cleric, Archbishop of York George Sentatmu, has slammed the centre.
"To hold someone for up to four years without charge clearly indicates a society that is heading towards George Orwell's 'Animal Farm'," Archbishop Sentamu said.
Hicks’ lawyer presses for review
Among the centre’s 500 or so inmates is Australian, David Hicks, who has been kept at Guantanamo Bay without trial for more than four years.
His lawyer, David McLeod, is pressing the Australian government to review its position on his client in light of the new UN findings.
He told ABC radio he met foreign minister Alexander Downer where he presented him with new information regarding David Hicks’ case and was advised the minister would take it to the government’s next cabinet meeting.
“The government has indicated a preparedness to have an open mind on this and whether it’s prepared to change its view remains to be seen,” Mr McLeod told the ABC.
“I want to impress upon Mr Downer the importance of making robust representations to the Americans, because I believe if the Australian government did make robust presentations, that David would be released back to Australia.”
US Supreme Court case opens
Meanwhile, the US Supreme Court has begun reviewing a crucial test case that may challenge the legality of the special military tribunals set up to try those detainees who have been formally charged.
So far there are nine detainees, including David Hicks, who have been charged.
Lawyers for Salim Ahmed Hamdan, a Yemeni accused of being a driver for al-Qaeda chief, Osama bin Laden, are contesting legislation which has barred detainees from being tried by a regular US military court martial.
A law passed by Congress in December denies detainees access to US courts through habeas corpus – the right to press for release from unlawful detention.
The Supreme Court is expected to make its decision next week.