Around 20 people who have had their passports cancelled are launching an appeal against the decision.
Their lawyer Zali Burrows says they all received letters saying they've had an adverse security assessment on grounds they're likely to engage in conduct that might prejudice Australian security.
One of the group, a 19 year old Australian citizen known only as Abu Bakr, denies he had any travel plans and is yet to surrender his passport.
He says he's risking a prison term by refusing to surrender the document.
"I haven't done anything wrong", he told SBS.
"I'm an Australian citizen..my record is good as gold."
Abu Bakr says he regularly preaches his faith and travelled to Egypt last year.
He was involved in 'Prophet Mohammed' protests in Sydney this year - but he says he was not one of those arrested.
"Where's the values, where's the human rights?", he asks.
ASIO's move comes after Australian Federal Police (AFP) and NSW Police last week charged two men for allegedly supporting Australians fighting in Syria's civil war.
In the letters, the men were accused by ASIO of having a 'jihadi mentality'.
"It seems ASIO has conducted an investigation without them knowing" said Ms Burrows.
"They haven't had the right of reply".
She's also representing 39 year-old Hamdi Alqudsi who was arrested last week and accused of assisting Australians to travel to conflict-ridden Syria.
ASIO says the agency can request passport cancellations on security grounds.
"Withholding passports is an important means of preventing Australians from travelling overseas to engage in activities prejudicial to national security - for example, to train, support or participate in terrorism," the spokesperson said in a statement.
"It may also be used to help prevent an Australian already overseas from participating - or further participating - in activities that are prejudicial to the security of Australia or another country."
Last week, Sydney man Hamdi Alqudsi fronted court for allegedly organising for Australians to engage in Syria's civil war.
Alqudsi was the alleged linchpin in a criminal group recruiting Australian men to fight with terrorist groups Jabhat al-Nursa and al-Qaeda affiliates.
The 39-year-old allegedly organised travel and contacts for seven Australians, including Amin Mohammed, who was arrested at Brisbane Airport on December 3.
