Japanese broadcaster NHK reported on Monday that he died last week following heart failure.
Yamada, known to his Japanese compatriots as "Olympic Ojisan", or "Olympics Grandad", first experienced the Games when Tokyo last hosted the gathering in 1964.
He had been a colourful presence at every Summer Games since, in his distinctive gold top hat and red jacket to pair with his beaming smile.
In an interview with Reuters in October, Yamada had expressed his desire to live long enough to see the Tokyo 2020 Games.
"It will be the culmination of all my years cheering the Olympics," Yamada had said.
Yamada’s haul of flags, stamps, photographs and other items collected on his Olympic travels are on display at a gallery in his hometown of Nanto City, Toyama Prefecture.
(Reporting by Jack Tarrant; Editing by Amlan Chakraborty)