Infections by the mosquito-borne Zika virus in pregnant women can cause microcephaly, a birth defect marked by small head size that can lead to severe developmental problems in babies.
The connection between Zika and microcephaly first came to light last fall in Brazil, which has confirmed more than 1,400 cases of the condition.
"Although the risks associated with the Zika virus can be minimal and precautions can be taken, my wife Jessica is pregnant, and I don't want to risk bringing anything back that could potentially have an effect," Van Garderen said in a statement through USA Cycling.
"If circumstances were different I would have loved to be selected again to represent the USA, but my family takes priority and it's a decision that I'm completely comfortable with.
"I hope that I'll be in the position to race at the 2020 Olympic Games."
Van Garderen's decision comes a week after more than 100 medical experts, academia and scientists called for the Rio Olympics to be postponed or moved because of fears that the event could speed up the spread of the Zika virus.
Their assessment countered the view of some leading experts of infectious disease who say that as long as the necessary precautions are taken there is no reason to cancel the Games.
Van Garderen competed in the road race at the 2012 Olympics, finishing 104th after working for team mate Taylor Phinney.
The Rio course would have suited him better as it contains a major climb, and he stood a strong chance of being selected for one of two time trial spots for the U.S. team, according to Cyclingnews.
Van Garderen will still have a busy schedule as he plans to ride in the Tour de France and Vuelta a Espana, but said he will not compete in the world championships in Qatar.
The World Health Organization has also said there is strong scientific consensus that Zika can also cause Guillain-Barre, a rare neurological syndrome that causes temporary paralysis in adults.
Former world number one golfers Adam Scott of Australia and Fiji's Vijay Singh, and fellow major winners Louis Oosthuizen and Charl Schwartzel of South Africa, are among those who previously announced they would skip this summer's global sports showpiece in Rio de Janeiro.