Australia trailed for most of the match after a disputed Japanese goal in the 23rd minute of the opening Group F game.
But with ten minutes to go, Tim Cahill slammed home amid a goalmouth scramble following a Lucas Neill long throw-in to break the 1-1 deadlock.
Then Cahill struck a second in injury-time with a long-range shot which cannoned off the post and into the net.
John Aloisi tadded a third goal minutes later to seal Australia's win in its first World Cup appearance since 1974.
Australia now plays red-hot favourite Brazil in its next group match in Munich on June 18, before Croatia in Stuttgart on June 22.