The pair were prevented from boarding a flight home so they could answer questions about a robbery that happened over the weekend.
Fellow swimmer James Feigen was also detained for questioning.
On Sunday, gold medalist Ryan Lochte and the three other swimmers said they were robbed at gunpoint while traveling back to the Olympic village in a taxi after a party.
Lochte said armed men carrying police badges pulled them over, ordered them to drop to the ground, and demanded their wallets and belongings.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
Police said they have been unable to find the taxi driver to corroborate the story.
The judge said there were inconsistencies between Lochte's story, who said only one man was involved in the robbery, and James Feigen who said several men had robbed the group.
Police said 32-year-old Lochte had already flown home on Monday, before the judge had ordered the four swimmers' passports to be seized.
Police sources said the three men currently being held are not facing any charges.
