Witnesses said the cameraman, Martin Adler, was filming the protest led by the Islamic Courts Union at the front of a crowd of thousands.
"A gun went off, he went down, that was it," Guardian correspondent Xan Rice, who was present, told Reuters.
"There was mass confusion. We were shunted to the side, and the rally was called off."
"The man was in a vehicle and came out to take video shots of some angry youths who were burning US and Ethiopian flags ... it was a single shot and within a second he was down," another witness said.
The Swedish foreign ministry said Adler, who covered more than two dozen war zones in his career and was married with two daughters.
Adler worked as a freelance reporter, photographer and cameraman for, among others, Swedish tabloid Aftonbladet and Britain's Channel Four television, his employers said.
The shooting came hours after the Islamists and the government signed a deal in Khartoum aimed at preventing confrontation and starting negotiations.
Since the Islamists took over, several Western journalists have gone into the city - previously too dangerous to visit - at the invitation of the ICU, who say their sharia courts have brought peace and order to a country in desperate need of it.
The shooting is at least the 10th killing of a foreign journalist in Somalia since 1991 and the first since BBC producer Kate Peyton was shot in May 2005.
The ICU blamed the shooting on followers of a lesser-known warlord, Abdi Awale Qaybdiid, who has remained in Mogadishu despite the Islamists' takeover.
Chairman of the ICU, Sheikh Sharif Sheikh Ahmed, condemned the killing.
"It is barbaric and we will punish those responsible," he told AFP.