Lloris's poor clearance fell straight to the midfielder who lobbed the keeper from inside his own half to steal the points and delight the home fans and their celebrating players.
France had led through a superb left-foot strike by forward Olivier Giroud after 37 minutes but that was cancelled out by a precise low finish from Sweden's Jimmy Durmaz in the 43rd.
The win put Sweden top of the group, level on 13 points with France and three ahead of the Netherlands, who beat visiting Luxembourg 5-0 on Friday.
The Swedes, who had lost 2-1 when the sides met in Paris in November, spent the opening half-hour frustrating the visitors, getting stuck in to challenges and slowing the tempo, but that all changed when Giroud scored and the game burst into life.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
France forward Antoine Griezmann twice headed straight at Robin Olsen in the second half but went closer with a free kick from distance that the goalkeeper scrambled to parry as the beaten Euro 2016 finalists upped the tempo.
Minutes later Lloris was tipping a free kick from Emil Forsberg over the bar and Sweeden's Jakob Johansson also had a thumping shot from distance that flew just wide of the target.
France brought on teenage striker Kylian Mbappe late on but it was Swedish substitute Sebastian Larsson who pressured Lloris into making the bad clearance that gave Toivonen the chance to win the game for the home side, and he took it gleefully.
(Reporting by Philip O'Connor; Editing by Ken Ferris)
