Mr Biden picked Gen Austin, and the Senate endorsed him, despite a law that says the US military must be led by a civilian or, if a former military official, someone who has been out of the service at least seven years.
The requirement is to ensure civilian control of the military.
That meant both houses of Congress had to grant a waiver for Gen Austin, who retired in 2016.
Mr Biden chose a former officer he knew well from when he was vice president in the administration of Barack Obama.
Gen Austin, a West Point graduate who served four decades in the military, was commander of US forces in Iraq and then head of the US Central Command covering the Middle East, from 2010 to 2016.
The two bonded because Gen Austin was friendly with Mr Biden's late son when both were serving in Iraq.
Mr Biden called him "uniquely qualified to take on the challenges and crises we face in the current moment" when announcing the nomination in December.
Gen Austin, known to be media-shy and not particularly politically savvy, tweeted a statement quickly after gaining Senate approval.
"It's an honor and a privilege to serve as our country's 28th Secretary of Defense, and I'm especially proud to be the first African American to hold the position," he wrote.
"Let's get to work."
He honed in on two urgent issues facing the US military in his confirmation hearing on Tuesday.
He singled out China as the country's most potent adversary.
Under his leadership, the Pentagon would "make sure that we are prepared to meet any challenge and that we continue to present a credible deterrent to China or any other aggressor who would want to take us on, and convince them that would be a really bad idea," he said.
He also pledged to tackle extremists in the forces, after some members of the military took part in the 6 January attack on the US Capitol by supporters of then-president Donald Trump.
"The activity that we've seen recently in terms of potential racists or extremist behavior within our ranks is in my view absolutely unacceptable," Gen Austin said.
"The job of the Department of Defense is to keep America safe from our enemies. But we can't do that if some of those enemies lie within our own ranks," he added.