But Labor's Home Affairs spokesperson Kristina Keneally described Mr Kassam as a "career bigot", citing a series of offensive social media posts.
In one tweet, Mr Kassam said Scottish Party Leader Nicola Sturgen, who had suffered a miscarriage, should have her mouth and legs taped shut so she could not reproduce.
The 32-year-old also described a female politician as a "wrinkly old ginger bird" and suggested a former British minister "was in the special needs class" at school.
"Mr Kassam has an extensive history of vilifying people on the grounds of their race, religion, sexuality and gender," Senator Keneally told Parliament on Tuesday.
"He has described the Koran, the holy book of the Muslim faith, as fundamentally evil."
Senator Keneally also said Mr Kassam had campaigned for limited migration to control, what he has described as "large-scale Muslim immigration".
Senator Keneally called on Home Affairs Minister Peter Dutton to refuse "career bigots" a visa to visit Australia.
"Why is the Home Affairs Minister allowing this individual into the country?"
Mr Kassam said he was not phased by Senator Keneally's "lies" declaring he would still be coming.
"I got a massage and played ping pong instead," he tweeted.
"I'll see her in court if she repeats her defamation outside the chamber, where she currently hides behind parliamentary privilege."
It's the first time the Republican-aligned conference which attracts tens of thousands of people in the United States each year, is being held in Australia.
Conference on track to sell out
Liberal MPs Amanda Stoker and Craig Kelly will join former Liberal candidates Warren Mundine and Jacinta Price at the event sponsored by Liberty Works, the American Conservative Union and Advance Australia.
The speakers list includes Fox News commentator Jeanine Pirro, American Conservative Union chairman Matt Schlapp and NSW One Nation leader Mark Latham.
In an interview with SBS News, Ms Stoker defended her participation and that of other Liberals in the conservative conference.
"If we are doing our job properly as politicians we should be talking to people from all walks of life," she said.
"Trying to shame into silence anyone who would speak to a person who is wrong on an issue damages our capacity for constructive democracy."
Liberty Works founder Andrew Cooper said Senator Keneally's blast did not make sense.
"What she's done is called a Muslim man an Islamaphobe and she's accused Warren Mundine and Jacinta Price, who are both Indigenous Australians, she's essentially accused them of being racists.
Mr Cooper said Mr Kassam, a former chief adviser to former UKIP leader Nigel Farage, would have some interesting things to say.
"Brexit is a significant moment in world politics and I think it's important that we hear from the people that were intimately involved in those events.
"Raheem Kassam is a respectable, well-known, albeit potentially controversial in some aspects of what he says, but he's a respected speaker and he speaks all over the world."
Mr Cooper said Ms Keneally's attack would not hurt the three-day event which was on track to be a sell out at the 600-seat Rydges World Square.