Australia's team chef de mission Kitty Chiller told News Ltd media on Sunday that their Olympians would be barred from going to the urban slums, even on official tours, in accordance with advice from a security expert.
"There is still much unfamiliarity about Rio and Brazil," Paes told Brazilian media.
"There is a certain dramatization. And between us, the Australian committee has been a source of aggressions to Brazil."
Rio's favelas are among the most violent and crime-ridden areas of the city.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
Australian Olympic Committee (AOC) president John Coates, a member of the International Olympic Committee's coordination commission, said in April 2014 that Rio's Games preparations were the worst he had ever seen and critically behind schedule.
Coates was more generous in a January assessment, however, saying he felt "very positive" about the delivery of the Games.
AOC spokesman Mike Tancred was also upbeat about the preparations in a statement on Wednesday but reiterated that the favelas remained off-limits.
"We have a team of 450 athletes, there is no way we could manage, or police, visits to the favelas by our athletes," Tancred said.
"(Our security expert) has advised that it would be impossible for us to allow our athletes to visit the favelas because we could not control visits involving a large number of athletes going to different places at different times."
Tancred said Australian athletes would "certainly engage with the residents of Rio and they will join in the fun on Copa beach".
"But the favelas are areas we cannot control and the personal safety of our athletes must come first," he said.
(Reporting by Ian Ransom; Editing by John O'Brien)
