Prosecutors believe the men were sent by IS to Germany late last year to either carry out a mission or to wait for further instructions.
They are suspected of having links to the perpetrators of November's Paris attacks.
German authorities say the three Syrian men were arrested in raids as part of efforts to root out extremists sent to Europe amid the migrant influx.
The trio has been under observation for months.
A spokesman for the federal prosecutor, Doctor Stefan Biehl, says it is believed the men were sent to Germany in mid-November last year as a possible "sleeper cell" - intended to stay dormant in the community until they were called upon to perform an act of terrorism.
"According to our investigations we do not have an indication for any concrete mission on the part of the accused - our current assumption is that they have been here pending further instructions."
The BBC's security correspondent, Frank Gardner, says if German authorities are right about the trio being part of a sleeper cell it is a worrying development.
"Because people who have done time in Syria are by nature going to be that much more dangerous if they are terrorists because they will have military training, or at least guerilla training. A more serious category of threat. I have to say that if they are indeed a sleeper cell and they have kept quiet and kept their noses clean (out of trouble) for nearly a year, that is a new method of operation."
German authorities say the men, aged 17, 18 and 26, travelled to Germany via Turkey and Greece posing as asylum seekers.
The route was used by most migrants to Europe last year.
Germany's Interior Minister, Thomas de Maiziere, says the trio were probably helped by the same network that funnelled I-S militants into the French capital, Paris, to carry out shootings and bombings that killed 130 people in November last year.
"It is wrong to hold refugees under general suspicion. But the security services always have tip-offs that individual potential terrorists or terrorist sympathisers could also be among the refugees. The authorities respond to every lead very carefully and quickly, and most of them have not proved to be true. We are talking at present of around 60 cases being investigated, and that's among many hundred thousands of newly arrived people."
Germany's open-door policy saw around a million refugees arrive in the country last year.
Concern about their presence has grown since the attacks, raising pressure on German Chancellor Angela Merkel to limit arrivals to 200,000 refugees per year.
She refuses to set such a limit.
