He said that concern prompted him to represent his constituents and attend a Melbourne rally organised by far-right ultra nationalists, the United Patriots Front.
Senator Anning ultimately wants a ban on Muslim and Sudanese migrants coming to Australia.
"I would not bring any more Muslims or Sudanese in the country," he told the Nine Network's Today Show.
"I would put a ban on that. And if any of them committed a crime, I would be shipping them home to where they came from."
Members and supporters of the United Patriots Front group were filmed performing the the "Sieg Heil" Nazi salute during the rally at St Kilda beach on Saturday.
Three people were arrested by police. Politicians from the different parties have condemned Senator Anning's actions, including his intention to charge taxpayers $2852.80 for his flights to the Saturday event.
Mr Coleman said the federal government was not about to take on board Senator Anning's suggestion.
"We welcome people from all backgrounds in Australia. We have a non-discriminatory migration policy," he said.
"We have people in Australia from all corners of the globe...People are welcome to Australia, so as long as they are willing to abide by Australian values, by the rule of law, and indeed to behave in a peaceful way."
He said if people broke the law in Australia, there would be consequences, regardless of their cultural background.
"If people commit crimes regardless of their cultural background. It doesn't matter what background someone is from. The law applies equally to everyone. That is a fundamental principle of our society."
Mr Coleman also condemned Senator Anning's attendance at Saturday's rally as "absolutely morally wrong", while the Queensland premier urged voters to ensure he is not re-elected.
"For a group of people to behave in this way in St Kilda and to make use of Nazi salutes and other imagery, it is disgraceful," the minister told reporters in Newcastle on Monday.
"Fraser Anning should absolutely not have attended that rally."
'Vote him out'
Queensland Premier Annastacia Palaszczuk called for voters to 'get rid' of Senator Anning and vote him out of office at the federal election in May.
"As the granddaughter of Polish migrants who made Australia home after fleeing the brutality of the Nazi regime, I am appalled that an elected representative would attend such a rally," the premier said in a statement on Monday.
Earlier on Monday, Senator Anning conducted a round of breakfast television interviews, defending his actions, insisting he is not a racist.
"There was no racist rally," he told the Nine Network, while also insisting he is not a racist himself.
"There were decent Australian people who demonstrated their dislike for what the Australian government has done which has allowed these people to come into this country and then bash people at random on the beaches, in their homes."
'Nothing to do with racist remarks'
He also defended the people he stood beside during the rally, saying they were not involved in doing Nazi salutes.
Senator Anning is adamant he was among ordinary, working Australians at the rally he attended, rather than radicals or skin heads.
"There were no Nazi salutes and there was nothing to do with Nazi racist remarks," he told ABC Radio National's AM program.
He said his evidence that there was a problem with "Sudanese gangs" comes from the media and some police reports and that he would protest against any criminal gangs.
Queensland Police denied there was such a gang issue in the state. They said they had not revised their position set out at a press conference from December, following a stabbing and armed robbery allegedly involving men of African appearance.
"What we're seeing in terms of these young African men, is we're not seeing gangs," Detective Superintendent Tony Fleming told reporters at the time.
"We want to make it really clear we're not targeting the whole African community, this is all about behaviour and those who are committing those offences."
- with AAP