Security forces said the bodies were found in a village to the north and that they had been blindfolded, bound and shot in the back of the head.
Major General Ghassan al-Bawi, the police chief of Diyala province, said 10 of the drivers were Sunni, the rest Shi'ite.
As Mr Rumsfeld held talks with Iraqi leaders on the escalating sectarian violence in the country, Prime Minister Nuri al-Maliki told parliament a national reconciliation plan he has promoted was Iraq's "last chance" to stem the violence.
"If it fails, I don't know what the destiny of Iraq will be," he said.
Mr Maliki has offered talks with some Sunni rebels and a limited amnesty under his 24-point plan in a bid to draw Sunnis, the seat of the insurgency, closer into the political process.
Several hours after Mr Maliki spoke, clashes erupted between gunmen armed with rocket-propelled grenades and police and residents in Um al-Maalif, a mainly Shi'ite neighbourhood in southern Baghdad.
Police said at least two people were killed.
Meanwhile US commander in Iraq, General George Casey, said Sunni militants in al Qaeda were stoking the sectarian violence that pits newly empowered majority Shi'ites against the once-dominant Sunni Arab minority of Saddam Hussein.
"What we are seeing now as a counter to that are death squads, primarily from Shi'ite extremist groups that are retaliating against civilians," he told reporters.
"So you have both sides now attacking civilians. And that is what has caused the recent spike in violence here in Baghdad."
The US ambassador said communal bloodshed was now a bigger threat than al Qaeda.
Mr Maliki said Iraqi security forces had defeated a coordinated attempt in recent days by gunmen to occupy Baghdad districts west of the Tigris.