A police official said two men became embroiled in a row in the market when one man hit the other in the chest, who then responded by lobbing a hand grenade into the air.
Witnesses said the grenade landed on a corrugated roof and exploded, critically wounding six and officials fear that the death toll could rise.
Security forces sealed off the area and launched a hunt for the man who fled after throwing the device.
The market, in the working class southeastern district of Shumaila, is a trading place where merchants sell Qat, a highly popular mild narcotic leaf that is legal in Yemen.
Authorities have blamed the incident, on the widespread use of illegal weapons in Yemen.
Yemen, Al-Qaeda leader Osama bin Laden's ancestral homeland, has seen several militant attacks in recent years. The most notable attack was the bombing of US destroyer Cole in the southern port of Aden in 2000 that killed 17 American sailors.
A number of terrorism-related cases involving Islamist militants are currently under way in Yemen, one of the poorest countries in the world.
Meanwhile, many government and aid officials are pushing to cut the use of the rubbery green leaf with amphetamine-like qualities.
It is blamed for many of this country's woes, from widespread poverty to growing health problems.