Despite the result and a dangerous touch of wheels in the sprint, Sagan held on to the overall lead at the end of the 193km stage from Rheinfelden to Champagne.
But the glory went to Richeze, a rider usually working in the service of others, for bagged a rare major victory with the selfless assistance of Gaviria.
“What a race this was, I still can’t believe it! It’s like a dream," Richeze said.
"This morning, at the briefing, we talked about the finale, because we knew the tricky and technical corner will be the key to winning the stage and our plan was to be first and second as we entered in that bend.
"My sprint was until the corner, but as Fernando didn’t try to overtake me, deciding to protect me instead, I went all the way to the line.
"I want to thank him for this and also to the entire team for their huge work and support.”
Matt Brammeier (Dimension Data), Jeremy Maison (FDJ), Lukas Jaun (Roth) and Nick Van der Lijke (Roompot-Oranje Peloton) quickly established a 3min 20sec lead on the peloton but were not allowed to add time to their effort.
Etixx-QuickStep joined forces with Tinkoff at the front and with 50 km to go, the gap was brought down to two minutes. As they entered Champagne to take on the last 20 km, the gap was still hovering just under the two minutes mark.
But with more teams contributing it was all back together again with 4km to go.
Diving into the last corner, with just 150 meters to go, Danny Van Poppel (Sky) nearly lost control but managed to stay upright to avoid a crash.
In front, Richeze was leading out Gaviria but instead of passing his teammate, Gaviria stayed on the wheel and celebrated as his designated leadout rider took his first win since 2012.
Sagan finished third and thereby increased his overall lead to Jurgen Roelandts (Lotto Soudal) and Silvan Dillier (BMC).
The general classification riders will finally get their chance to shine on Stage 5 with a 125km mountain stage stage from Brig-Glis to Cari.