She beat the incumbent, Jean Ping of Gabon, in a closely fought election over several rounds of voting.
"Now we have the African Union chair Madame Zuma, who will preside over the destiny of this institution," Benin's president and current AU chairman Thomas Boni Yayi said.
Dlamini-Zuma, 63, an experienced diplomat, is a veteran of the fight against apartheid. A doctor by training, she has served as health, interior and foreign minister in South Africa.
Her victory brings to an end an impasse that has lasted for the past six months.
Her former husband, South African President Jacob Zuma, was one of the first to offer his congratulations after the vote.
"It means a lot for Africa... for the continent, unity and the empowerment of women -- very important," Zuma said.
Voting took place at an AU summit, where, earlier in the day, Rwanda and the Democratic Republic of Congo agreed to an international force to neutralise rebels in eastern DR Congo, as the African Union said it was ready to send peacekeepers there.
Rwandan President Paul Kagame told AFP both sides had agreed "in principle" to accept the force.
He was speaking after his first face-to-face meeting with DR Congo President Joseph Kabila since a UN report in June accused Rwanda of supporting Congolese rebels. The two leaders met on the sidelines of the AU summit.
Dlamini-Zuma's win follows her challenge six months ago to unseat Ping, the former commission chairman, which ended in deadlock after neither won the required two-thirds of the vote, leaving Ping in the post.
"She's a freedom fighter, not a bureaucrat or a diplomat," said Ugandan President Yoweri Museveni, beaming enthusiastically.
Noureddine Mezni, spokesperson for the outgoing chairman, told AFP that Ping had acknowledged defeat.
He "has accepted the results of the elections and wishes Madame Dlamini-Zuma the very best.
"He expressed his readiness to cooperate with her to work together for the unity of the continent."
Erastus Mwencha of Kenya was re-elected as deputy chair of the AU Commission, he added.
Members of the South African delegation smiled and congratulated one another as they filed out of the conference hall.
"It's good for southern Africa. We (in southern Africa) never had this job," a delegate from Zimbabwe told AFP with a broad grin.
Officials said the elections went to four rounds of voting before Dlamini-Zuma won 37 votes, three more than the required majority, to confirm her win over Ping.
Jakkie Cilliers of the South Africa-based Institute for Security Studies told AFP how Dlamini-Zuma's score had crept up from one round of voting to the next.
"She got ahead in the first round and after that the momentum kicked in," said Cilliers. "The heads of state wanted a decision."
Dlamini-Zuma's win had brought "clarity as to who's in charge" at the AU, after six months of deadlock over the leadership issue, he added.
Some analysts say South Africa has violated an unwritten tradition that continental powerhouses do not run candidates for the post, but leave smaller nations to take the job.
Before the vote however, Dlamini-Zuma played down concerns that the vote could divide the AU.
"I don't think the continent will be polarised," she said.
The winner would "make sure they work with everybody, irrespective of where and who they voted for," she added.
Cilliers said he was optimistic that divisions created by the vote would heal.
"She'll be an inclusive chair, she won't be divisive ... She was a very competent foreign minister and even better at home affairs (the interior ministry)," he said.
Rwandan and DRC leaders will meet again in early August to try to thrash out the details of the force -- including size, mandate, nationality and deployment details -- for eastern DRC.
UN peacekeepers already operate in the region. UN deputy secretary general Jan Eliasson called for an immediate end to the violence, warning that "countries of the region ought to respect the principle of non-interference."
AU Peace and Security Commissioner Ramtane Lamamra said that changing the mandate of the UN peackeeping mission there "was one scenario."
On Monday the heads of state will wrap up their two-day meeting.
