Gates, 63, served as CIA director from November 1991 to January 1993 during the presidency of Republican George Bush, the current president's father.
He joined the CIA in 1966 and spent nearly 27 years as an intelligence professional. He was also a member of the national security council.
Currently Gates is the president of Texas A and M University in College Station, Texas, and is a member of the bipartisan Iraq Study Group. That group, headed by Republican former secretary of state James Baker -another Texan - and Democrat Lee Hamilton, is working on recommendations to present the government about the Iraq war.
"I've asked Bob Gates to serve as the secretary of defense," Bush said one day after opposition Democrats took firm control of the US House of Representatives.
"If confirmed by the Senate, Bob will bring more than 25 years of national security experience and a stellar reputation as an effective leader with sound judgment, "he said, noting that he "rose from an entry level employee in the CIA to become the director of central intelligence."
During his service Gates "gained first-hand knowledge that will help him meet the challenges our country faces during the next two years."
Bush described him as "a steady, solid leader who can help make the necessary adjustments in our approach to meet our current challenges."
Gates replaces the controversial Rumsfeld, who Bush praised as "a superb leader during a time of change," a "patriot who served our country with honour and distinction," and a "trusted adviser and friend."
The incoming speaker of the House, Democrat Nancy Pelosi, earlier in the day called for Rumsfeld's resignation over his leadership in the Iraq war.