Police accused the 27-year-old Bozhilov, who finished ninth in the rowing men's double sculls with team-mate Kristian Vasilev, with striking the cleaners with a broomstick, punching them and choking one woman.
Bozhilov and the Bulgarian delegation denied there had been any physical confrontation but said they had distrusted the local staff in the Olympic village because they had repeatedly been the victim of thefts of their personal belongings.
Bozhilov said he and Vasilev had told the women cleaners to leave because they did not want them there.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
"Several maids were sitting in our room, it was obvious that they were not preparing to clean, and I told them to leave," Bozhilov told Bulgarian Nova TV shortly after arriving back.
"They speak only Portuguese and we tried to show them we wanted them to leave the room, using gestures."
He denied using any physical force.
"A short time after that, one of the women complained to the block administration that she had been 'subject to aggression' which did not correspond to the truth," the delegation said in a statement.
Bozhilov and Vasilev were not arrested and were allowed to return home.
"There are many guards all over the village but all the same a lot of things were disappearing from our room," Bozhilov said. "Everyone in the village was talking about theft. There are many organisational problems there."
The Bulgarian delegation pointed the finger at local staff.
"Considering the actual circumstances, the constant danger of theft and the reluctance of athletes to be disturbed, the leadership of the Bulgarian Olympic delegation finds such behaviour by staff extremely inappropriate and speculative."
Bulgaria's sports minister, Krasen Kralev, said on Monday that he had been robbed of his watch while entering the Olympic athletics stadium.
(Reporting by Angel Krasimirov; Editing by Richard Balmforth)
