The assault was one of the deadliest against Yemeni troops and the latest in a spate of attacks on security forces since President Abdrabuh Mansur Hadi pledged a crackdown on the militants in an inauguration speech last month.
A medical official at the military hospital in the southern port city of Aden, speaking on condition of anonymity, said "the death toll... has risen to at least 103" soldiers.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
He said "many soldiers died from wounds sustained in the assault" on army posts on the outskirts of Zinjibar, Abyan's provincial capital where militants linked to al Qaeda are in control.
A military official, who also declined to be identified, told AFP al Qaeda militants were responsible for the "surprise attack" which had turned into "a massacre."
Another medic said hospital staff were overwhelmed by the number of casualties.
"We were forced to use administrative offices and waiting rooms to treat the wounded," he told AFP, declining to be named. "The hospital was packed full with dead and injured" soldiers.
In Washington, a Pentagon spokesman said the United States is "very concerned" about the assault.
"We view Yemen as a very important partner on counter-terrorism efforts and we're also very concerned about the clashes that have taken place there, to include AQAP (al Qaeda in the Arabian Peninsula) advances in certain parts in the country," press secretary George Little told reporters.
"I think it's important to put this into some context. The Yemeni government has faced challenges in certain parts of the country for some time so I wouldn't necessarily read anything at this point into the stability of the Yemeni government," he said.
Yemeni military officials reported fierce clashes on Sunday when suspected al Qaeda militants tried to overrun an army post in Kud, just south of Zinjibar.
The violence then spread to other military positions on the outskirts of the city.
At least 25 al Qaeda gunmen were killed in the fighting and several others wounded, a local official from the nearby militant stronghold of Jaar told AFP.
He also said at least 56 soldiers were captured by al Qaeda, including seven officers and 10 wounded soldiers.
The militants, known in Yemen as the Partisans of Sharia (Islamic law), seized control of Zinjibar and several other towns in Yemen's mostly lawless south last May as former president Ali Abdullah Saleh faced mass protests.
The military official, who was at the scene during Sunday's attack, said troops from the Kud base were "surprised" to see the militants carrying army issue weapons and using military vehicles.
Soldiers who survived the attack accused some army leaders who had served under Saleh of "collaborating" with al Qaeda.
On Monday, gunmen opened fire at the police chief of the Sheikh Osman district of Aden, Colonel Abdullah al-Mawzaie, wounding him and a companion as they headed for the southern province of Lahij, a security official said.
The violence highlights the security challenges facing Hadi as he tries to restore order and unify the country's armed forces, as stipulated by a Gulf-brokered transition accord that ended Saleh's 33-year rule.
On Friday, Hadi, who will lead Yemen for an interim two-year period, named General Salem Ali Qatan to head the 31st Armoured Brigade in southern Yemen, replacing Saleh loyalist General Mahdi Maqola.
The appointment was one of Hadi's first steps as head of a new military commission tasked with restructuring Yemen's divided security forces.
Some of Yemen's most powerful army units, including the Republican Guards, are commanded by Saleh's closest aides, including his son and nephew.
Yemeni political analyst Majed al-Mabjahi said Sunday's suspected al Qaeda assault was "a show of force" by the group.
"Given the new political environment, and the fear of impending attacks" by the new government, al Qaeda wants to demonstrate that any attempt to destroy the group would "come at a high cost."
In his February 25 inaugural speech, Hadi vowed to fight al Qaeda and restore security across the impoverished nation.
EU foreign policy chief Catherine Ashton pledged firm support to Hadi as she spoke to him by phone for the first time since his inauguration.
A statement, which did not refer to Sunday's deadly assault, said she had told him that "this is a key moment in Yemen's transition."
