Patrick Reimer scored the game winner a minute and a half into the extra period.
The win means that Germany will contend for a medal which would be their first in more than four decades.
Sweden had beaten Germany 1-0 in the preliminary round last week.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
Since German unification in 1990, the Germans have never beaten Sweden in Olympic play, going 0-5. Even before that, the best result a German national team had earned against the Swedes in the Olympics was a 1-1 tie in 1984 at Sarajevo. East Germany were also winless, with a 0-1 record.
The last time a German team won a medal at the Olympics was 1976 in Innsbruck, Austria, when they won bronze.
Germany will take on Canada in the semi-finals.
(Reporting By Dan Burns; Editing by Christian Radnedge)
