More than 200 police took part in the pre-dawn raids in northern Germany to detain the men, who are suspected of either plotting an attack or awaiting orders to commit one.
The men were identified only as Mahir al-H, 17, Ibrahim M, 18, and Mohamed A, 26.
Last October, they left Syria and travelled via Turkey and Greece -- the main route used by tens of thousands of refugees and migrants -- and arrived in Germany in mid-November.
Mahir al-H. had joined IS in its de facto capital of Raqa, Syria by September 2015 and received some weapons and explosives training.
The following month, all three had pledged to travel to Europe in talks with an IS member who was "in charge of missions and attacks" outside the Syria-Iraq region where the group has its self-proclaimed caliphate.
Related reading
Syria truce brings 'significant drop' in fighting
In Europe, "the three accused were meant to either execute a mission or await further instructions," the prosecution service said in a statement, adding that no evidence of "concrete orders or instructions" had been found.
The men had received fake passports, mobile phones and a four-figure cash sums in US dollars.
They were detained at three residential locations in the northern state of Schleswig-Holstein by more than 200 commandos of the federal police, federal criminal office BKA and police forces of several states.
Police also raided three asylum seeker shelters, Die Welt daily said.
Warrants for their arrest had been issued by a federal judge on September 7, based in part on intelligence provided by Germany's domestic security agency, the Federal Office for the Protection of the Constitution.
In July, Germany suffered two bloody attacks claimed by IS, which were carried out by migrants.
German authorities have urged the public not to confuse migrants and "terrorists," but have acknowledged that more jihadists may have entered the country among the one million asylum seekers who arrived last year.
