The comments were made in interviews in December for a new book written by journalists from The Australian newspaper to mark his 10th anniversary in power next month, and on Monday Mr Howard said he stands by his comments, saying sections of Australia's Muslim population are antagonistic to Australian culture.
He later defended his comments, telling reporters it is his "right and duty" to express his thoughts.
Spokesman for the Islamic Council of NSW Ali Roude said Mr Howard has a right to his personal view, but he should involve the entire community if he is considering a change to Australia's policy of multiculturalism.
"If the PM has a personal preference for assimilation rather than the strategy of multiculturalism which has been the strong bipartisan position in Australia since the days of the Fraser government, that is his personal right and he is entitled to it," Mr Roude said.
"I myself and our council will disagree with him but we will take it no further."
Lebanese Muslims Association spokesman Keysar Trad said Mr Howard is "unfortunately just pandering to the Islamaphobia out there by making these comments".
However the president of Sydney-based Islamic Charity Projects Association, Ghayath Al-Shelh, said he agreed with Mr Howard that the problem of extremism in Muslim societies needs to be addressed.
The president of the Australian Federation of Islamic Councils, Ameer Ali, said the Muslims referred to by Howard represented only a "tiny fraction" of the community.
But he added: "There is also a tiny fraction of Australians who believe in white supremacy"
The book -- The Howard Factor -- quotes him as saying that while he is confident the overwhelming majority of Muslim immigrants would be successfully integrated, there were unprecedented problems.
"I do think there is this particular complication because there is a fragment which is utterly antagonistic to our kind of society, and that is a difficulty," Mr Howard said.
"You can't find any equivalent in Italian, or Greek or Lebanese or Chinese or Baltic immigration to Australia. There is no equivalent of raving on about jihad (holy war), but that is the major problem," he said.
"I think some of the associated attitudes towards women are a problem. For all the conservatism towards women and so forth within some of the Mediterranean cultures, it's as nothing compared with some of the more extreme attitudes."
On Monday, Mr Howard said he stands by his comments that a small section of the Islamic population's views about Australia indicate an extremist view.
"It is not a problem that we have ever faced with other immigrant communities who become easily absorbed by Australia's mainstream.
"We want people when they come to Australia to adopt Australian ways.
"We don't ask them to forget the countries of their birth, we respect all religious points of views and people are entitled to practise them but there are certainly things that are not part of the Australian mainstream."
He underlined that he made the comments before race riots erupted at Cronulla beach on December 11.
"I was not trying to make some kind of tawdry political point, it is a view that I have held for some time," he said.
There are around 300,000 Muslims out of Australia's overall population of 20 million.