"The military command of Shebab mujahedeen ordered a tactical retreat at midnight," Shebab spokesman Ali Mohamud Rage told AFP.
The announcement came a day after an assault on the city by African Union troops, who had been trying to dislodge the insurgents from the key coastal city for days.
Residents in the city confirmed that the Islamist fighters seemed to have moved outside city lines and that their radio station, Radio Andalus, was off the air.
"We don't know where they went to... but early this morning the last military vehicle left the town," said Hassan Ali, a resident.
"Even their radio station is off air," he added.
Shebab fighters on the ground also confirmed the withdrawal.
"We got orders from our superiors to withdraw from the city... this is part of broader military tactic we have set for the enemy," said Sheikh Mohamed Abu-Fatma, a Shebab commander, told AFP by telephone.
Kismayo has been a vital lifeline for the Shebab since the African Union force in Somalia (AMISOM) force reconquered most of Mogadishu last year and Ethiopian troops stripped them of other key cities in the east.
Observers have consistently said that the loss of Kismayo would leave the Shebab, who once controlled 80 percent of the country, unable to retain large swathes of territory.
But in places where the Shebab have abandoned fixed positions, most notably in the capital Mogadishu, they have switched to guerrilla tactics and remained a threat.
