Sydney-based Sheikh Taj Eldine Elhilali sparked condemnation across Australia with a sermon publicised last week, blaming women for sexual assault and describing "immodestly dressed" woman as meat.
Amnesty's Secretary General Irene Khan, herself a Muslim, said that the Sheikh's comments reflected "mediaeval" thinking in which women were seen as sexual objects.
"I see it as a bigger issue of how men seek to control women's bodies, women's lives, what women wear," Ms Khan, who is in Australia to accept the Sydney Peace Prize, told ABC radio.
"What Alhilali doesn't recognise is that violence against women has less to do with what a woman wears and much more to do with the inequality, oppression, impunity and the apathy and tolerance that exists in society.”
"Of course, violence against women happens in non-Muslim societies as much as Muslim ones, so what you wear or what you don't wear, to me, is a red herring."
Ms Khan, who does not wear the Muslim hijab (headscarf), said a woman's clothing should be a matter of choice, not something a politician or religious leader governs.
"I think what they (leaders) should be doing is creating an environment in which every women can freely make that choice without threats or coercion or violence," she said.
Ms Khan rejected the notion that a piece of clothing would threaten diversity in communities.
"What they are doing by talking about a veil is pretending that a piece of clothing a woman wears is somehow preventing integration, is threatening multiculturalism. I find that actually quite ridiculous”, she said.