One of the protesters, a former CIA analyst, accused Rumsfeld in a question-and-answer session of lying about pre-war intelligence on Iraq.
"Why did you lie to get us into a war that caused these kind of casualties and was not necessary?" asked Ray McGovern, the former analyst.
He said Rumsfeld had told a "lie" about supposedly known caches of weapons of mass destruction in Iraq, and had falsely linked al-Qaeda to Saddam Hussein's regime.
"I did not lie," shot back Mr Rumsfeld, who waved off security guards ready to remove Mr McGovern from the hall.
With Iraq war support remaining low, it is not unusual for top Bush administration officials to encounter protests and hostile questions. But the outbursts Mr Rumsfeld encountered seemed beyond the usual.
Three protesters were escorted away by security as each interrupted Mr Rumsfeld's speech by jumping up and shouting anti-war messages.
Throughout the speech, a fourth protester stood in the middle of the room with his back to Mr Rumsfeld in silent protest. Officials reported no arrests.
Mr Rumsfeld also faced tough questions from a woman identifying herself as Patricia Roberts of Lithonia, Georgia, who said her son, 22-year-old Private Jamaal Addison, was killed in Iraq.
Ms Roberts said she is now raising her young grandson and asked whether the government could provide any help. Mr Rumsfeld referred her to a website listing aid organisations.
Mr Rumsfeld has been interrupted by anti-war demonstrators in congressional hearing rooms as he has delivered testimony to lawmakers in recent months, and at some speeches around the country.
Secretary of State Condoleezza Rice has had direct confrontations overseas. These include demonstrators who called her a murderer and war criminal in Australia in March, and throngs of anti-war protesters who dogged her every move in northern England in April.
More than half of Americans say the war in Iraq was not worth the cost financially or in loss of life, recent public polling has found. Just over one-third of those surveyed say they approve of Mr Bush's handling of the war.