The agreement comes despite reported violations at the start of a seven-day long ceasefire timed to coincide with the talks.
The prisoner swap involves some 360 Houthi rebels and around 260 civilians and fighters loyal to Yemen's President, Abd-Rabbu Mansour Hadi.
Officials from the two warring sides say the exchange is a confidence-building measure.
The mass prisoner swap is taking place despite both sides in Yemen's nine month-long civil war accusing the other of deadly violations of a ceasefire that's just two days old.
The seven-day truce coincides with United Nations-sponsored peace talks in Switzerland, aimed at clinching a permanent ceasefire and a political transition.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
Yemen's Ambassador to the UN, Khaled Al Yamani, has told Al-Jazeera the Yemeni government will stick to the ceasefire despite violations by Houthi forces.
"We think that this is the only option for us to go further down the path of peace and achieve a peaceful resolution to the conflict in Yemen."
This resident of the capital, Sanaa, is not optimistic about peace.
"I think the ceasefire will fail, just as the previous ones. I heard that they started to exchange fire only one hour after the ceasefire. But I still hope the problems can be solved."
Almost 6,000 people have been killed in Yemen's civil war.
