Patrick Reimer scored the game winner a minute and a half into the extra period.
The win means that Germany will contend for a medal which would be their first in more than four decades.
Sweden had beaten Germany 1-0 in the preliminary round last week.
Since German unification in 1990, the Germans have never beaten Sweden in Olympic play, going 0-5. Even before that, the best result a German national team had earned against the Swedes in the Olympics was a 1-1 tie in 1984 at Sarajevo. East Germany were also winless, with a 0-1 record.
The last time a German team won a medal at the Olympics was 1976 in Innsbruck, Austria, when they won bronze.
Germany will take on Canada in the semi-finals.
(Reporting By Dan Burns; Editing by Christian Radnedge)
