Mr Abe, who famously dressed up as the titular plumber from video game Super Mario at the Rio Games to represent Japan, played an outsized role in attracting the Olympics to Tokyo.
In a speech in front of a banquet room full of International Olympic Committee members in 2013, he said that the lingering nuclear disaster at Fukushima was "under control" and pitched his nation as a "passionate, proud, and a strong believer" in the Olympics.
At the time, Mr Abe and his supporters hoped the Olympics would parallel the 1964 Tokyo Games heralding the nation's revival after decades of economic stagnation and also mark its recovery from the devastating nuclear and natural disaster in 2011.
Instead, the Games, delayed a year because of the pandemic, has faced a series of setbacks, including the exit of Yoshiro Mori, the former head of the Tokyo 2020 organising committee, who resigned after making sexist remarks.
That was soon followed by the resignation of Tokyo Olympics creative head Hiroshi Sasaki after he made derogatory comments about a popular Japanese female entertainer.
The Tokyo organising committee said on Thursday Kentaro Kobayashi, who is listed as the show director for the Games' opening event, had been fired after a joke he made about the Holocaust as part of a comedy act in 1998 resurfaced in domestic media.
The opening ceremony on Friday, which normally stands as a major showcase of the host nation, is set to be subdued with Japanese media reporting that less than 950 people - including only around 15 global leaders - are set to attend.
US First Lady Jill Biden is expected to land in Tokyo on Thursday afternoon for the Games' opening ceremony, raising expectations she might also use her attendance to discuss vaccines with Prime Minister Yoshihide Suga.
Only a third of Japanese have had at least one dose of the vaccine, fuelling public concerns that the Olympics could become a super-spreader event. Already dozens of attendees have tested positive for COVID-19, forcing athlete withdrawals and teammates into isolation.
Mr Abe reportedly decided against attending the ceremony after the Japanese government declared a state of emergency and virus restrictions over Tokyo, in an effort to minimise health risks among residents and visitors.
Mr Abe's office could not immediately be reached on Thursday, a public holiday in Japan.
COVID-19 infections have jumped in the capital and are projected to spike further, straining healthcare providers.