The IOC said the athlete had tested positive for banned steroid turinabol and been disqualified.
He was one of four athletes named by the IOC as having tested positive for steroids in their re-tests from the Beijing 2008 Games and the London Olympics which in total yielded 98 positive results.
"Oleksandr Pyatnytsya, 31, of Ukraine, competing in athletics, has been disqualified from the Olympic Games London 2012 and ordered to return the silver medal from the javelin throw event," it said in a statement.
"Reanalysis of Pyatnytsya's samples from London 2012 resulted in a positive test for the prohibited substance dehydrochlormethyltestosterone (turinabol)."
The IOC stores samples for a decade to test with newer methods or for new substances and conducted targeted re-tests ahead of the Rio Olympics.
The IOC also disqualified Turkish weightlifter Nurcan Taylan from the women’s 48kg weightlifting event at the Beijing Games, where she did not finish. She won the gold medal at the Athens Olympics in 2004.
Armenian weightlifter Hripsime Khurshudyan, who won a bronze in London, was disqualified from the Beijing Games, where she was 11th, after testing positive for steroids as did Belarussian hammer thrower Pavel Kryvitski, who finished 28th in London.
