Schwartzel's decision has been announced by team captain Gary Player who said it weakened the country's chances of a medal in Brazil.
"I am sad and disappointed that several top players have withdrawn from the Olympic Games in Rio," Player said in a statement.
"I was sorry to hear that Louis Oosthuizen and Charl Schwartzel have withdrawn their names from consideration.
"I would have given anything to play in the Olympics. South Africa had a great team, but now obviously, it will not be as good."
Former Open winner Oosthuizen, 33, pulled out last week citing family commitments and a taxing schedule.
No reason was given for the withdrawal of Schwartzel, who has worked his way back up to 20th in the world rankings.
Golf returns to the Olympics after a 112-year absence, but its inclusion has been questioned by Australian swimming great Dawn Fraser who criticised compatriot Adam Scott's decision to skip the Rio Games.
Fiji's former world number one Vijay Singh has also opted out of the event.
The Rio Olympics take place from Aug. 5 to Aug. 21, with the men's golf tournament ending Aug. 14.
(Reporting by Mark Gleeson; Editing by Amlan Chakraborty)
