The veteran power broker's departure sent shockwaves though Washington, in a move that could possibly pave the way to a major change of US strategy in Iraq.
Bush named former CIA chief Bob Gates as the new man at the Pentagon, as Democrats celebrated capturing the House of Representatives for the first time since 1994, and moved to the threshold of power in the Senate.
The US president acknowledged that violence and bloodshed staining the US occupation of Iraq played a major role in the Democratic election victories.
"I recognize that many Americans voted last night to register their displeasure with the lack of progress being made there," he said.
"After a series of thoughtful conversations, Secretary Rumsfeld and I agreed that the timing is right for new leadership at the Pentagon," Bush said, adding that Rumsfeld would have left however the elections turned out.
Bush said he was "disappointed" at the result of the congressional elections and took responsibility, acknowledging the Democratic wave was whipped up by anger over the troubled US occupation.
"I'm obviously disappointed with the outcome of the election," Bush told reporters at a White House news conference.
"As the head of the Republican Party, I share a large part of the responsibility," Bush said.
"You look at it race by race, it was close. The cumulative effect, however, was not too close. It was a thumping."
The Republican drubbing means Bush is already certain to face new scrutiny over Iraq and a difficult two final years in the White House.
"The American people have spoken, and they have demanded change," said Senate Democratic leader Harry Reid. "They want, they deserve and they are going to get a new direction - at home and in Iraq."
"This is a wake-up call for the Republican Party," said a humbled Republican Senator John McCain, a likely candidate to replace Bush in 2008.
Democratic leaders quickly demanded a new direction to the US operation in Iraq.
Nancy Pelosi, the California representative set to become the first woman Speaker of the House, said the election proved Americans were sick of Bush's policies on Iraq, where more than 2,800 US troops have died.
"We cannot continue down this catastrophic path. And so, we say to the president, 'Mr. President, we need a new direction in Iraq,'" Pelosi told supporters.