Mr Rumsfeld's resignation was announced after a stinging vote of "no confidence" in legislative elections, which saw the Democrats take control of the House of Representatives and possibly the Senate.
Though Mr Rumsfeld had offered his resignation to the president twice, before and after the prisoner abuse scandal at Abu Ghraib in Iraq, President Bush stood by him, even as calls for his resignation mounted among Republicans.
Military resentment with Mr Rumsfeld boiled over into a public revolt by a string of retired former commanders who called for his resignation, but Mr Rumsfeld, a consummate bureaucratic infighter, survived that challenge.
His resignation marks the end of a remarkable career for the combative 74-year-old, who twice served as defence secretary in tenures that were 25 years apart.
Mr Rumsfeld was in the Pentagon on September 11, 2001 when airliners commandeered by hijackers flew into the World Trade Centre in New York and then the Pentagon, killing nearly 3,000 people in the deadliest attack ever on US soil.
In less than a month, he launched US forces in a war against al -Qaeda which toppled the Taliban regime in Afghanistan and mobilized the US military for a global war on terror.
Mr Rumsfeld pulled back US forces from Europe and repositioned them for a turbulent post Cold War era populated by suicide bombers, insurgents and "rogue" states armed with nuclear weapons.
In his nearly six years in office, Mr Rumsfeld has dominated the US defence establishment like few before him.
He's the oldest – and youngest -- person to serve as secretary of defence. He received the youngest title in 1975 when he took his first tenure at the age of 43.
The two stints have also made him the longest serving defence secretary after Robert McNamara, the Vietnam War-era defence chief.
Like Mr McNamara, Mr Rumsfeld was supremely self-confident, often abrasive, and not popular with the military brass, whom he prodded and poked into line and sometimes upbraided in public.
Americans embraced his tough-guy persona during the war in Afghanistan, but it faded in the lead up of the controversial invasion of Iraq to rid that country of weapons of mass destruction which later turned out not to exist.
Just weeks before the US invasion of Iraq, Mr Rumsfeld dismissed a warning by the army chief of staff that several hundred thousand troops were required to occupy Iraq, calling the estimate "far from the mark."
In remarks that would prove prophetic, Mr Rumsfeld told reporters it was "not knowable" how many troops would be needed.
"We have no idea how long the war will last. We don't know to what extent there may or may not be weapons of mass destruction used. We don't know – have any idea whether or not there would be ethnic strife.
"We don't know exactly how long it would take to find weapons of mass destruction and destroy them. There are so many variables that it is not knowable."
But Mr Rumsfeld said it was "not logical" that as many troops would be needed to pacify the country as to win the war and "any idea that it's several hundred thousand over any sustained period is simply not the case."
The United States invaded Iraq on March 20, 2003 with a force of fewer than 100,000 troops, but the failure to commit enough troops to secure the country is now widely viewed as a mistake, allowing a Sunni insurgency to take hold and setting in motion a spiral of violence that has seen no let up.
In Iraq, Mr Rumsfeld will be leaving a country that is on the brink of civil war, despite US efforts to build an Iraqi security force capable of stabilizing the country.