"I was just surprised, the stadium wasn't very loud at the time, it was just before a play of the game ... everyone could hear, it was baffling to me.
"You cop criticism as a player in general, fans try and put you off your game, that is part of the culture of the sport. But certainly not the remarks I received."
Western Sydney want to ban the spectator, who Young said was a male aged "50 to 60 to my eye".
"I was pretty displeased with the taste of the remarks," Young said.
"In society these days we have moved on from that sort of thing but unfortunately this happened.
"It personally didn't hurt me so much. I just thought it was poor form. A person his age and stature should know better.
"There's a lot of young kids around there and families at the game, you have got to understand that."
Young, a stalwart of the Roar since 2014, said he had previously encountered racism in his 15-year career.
"I have experienced that enough in my career ... racism is a very touchy subject for a lot of people," he said.
"Certain people have lost their lives through racism ... it's a very touchy subject."
Young accepted the Wanderers' apology after the incident at Glen Willow Sports Stadium, where Western Sydney and Brisbane drew 2-2 on Saturday.
"This incident is not a reflection of their supporters or the club, that is important to acknowledge," the 33-year-old Young said.
"It's just something that somebody said."
The Wanderers said the spectator's behaviour was "deplorable".
"This individual is not to be welcomed back to any Wanderers fixture," the club said in a statement.
Football Federation Australia is investigating the incident.