The 14-12 defeat at the Deodoro Stadium was one of biggest upsets in sevens rugby history and its impact on the All Blacks' campaign was compounded when Williams was forced off the pitch two minutes into the second half.
Williams, twice a World Cup winner in the 15-man game, partially ruptured his Achilles charging into a tackle and was replaced in the squad by Sione Molia.
In a third blow, Joe Webber was taken off the pitch on a stretcher with a dislocated left shoulder soon after Williams but, having already used the one replacement they are allowed, New Zealand have kept him in the squad.
"Devastating for Sonny, to lose him so early in the tournament with an Achilles problem and also Joe Webber," said New Zealand's veteran coach Gordon Tietjens.
"Both players really worked their butts off to make this tournament and both are influential players with outstanding qualities that I needed. But we've got to get on with it."
Japanese forward Lote Tuqiri said his team had received a "boost" in confidence from the departure of Williams as a large part of their gameplan had been designed around stopping the former rugby league international's trademark offload.
Japan are ranked 10th out of the 12 teams taking part in the Rio tournament but thoroughly deserved their first ever win over the All Blacks, who are seeded third.
In echoes of the victory of their 15-man team over South Africa at the Rugby World Cup last year, the Japanese showed no deference to the acknowledged superpower of the international game.
"It's unbelievable," said Japan back Lomano Lemeki. "You've got to think that a minnow team can come here and beat a team that's supposed to be a gold medal contender. It's still pretty unbelievable and I am just shocked. to be honest."
Teruya Goto and Kameli Soejima scored the tries for Japan and, crucially, Lemeki and Katsuyuki Sakai slotted the conversions.
That gave them the two-point lead they managed to cling on to for a famous victory, despite some desperate All Black attacks in the final couple of minutes.
New Zealand's day looked to be getting even worse when Collins Injera scored a try to give Kenya the early lead in their second Pool C match.
The All Blacks rallied for a 28-5 victory, however, which kept alive their hopes of qualifying for the quarter-finals after their final pool match against Britain on Wednesday.
"Down but never out, always praise to the most high for blessing me with so much! Let's go @nz7s we still in this," Williams later tweeted.
On Tuesday, New Zealand lost to Australia in the final of the women's tournament.
(Editing by Ed Osmond and Meredith Mazzilli)
