Ange Postecoglou kept faith with the more attacking 3-4-3 formation, but chances were few and far between in a nervy encounter between two giants of the Asian section.
Aaron Mooy's absence was clearly felt, with Australia looking a level below their hosts and while the opening stages were relatively even, Japan's superiority eventually saw them come out on top.
The Blue Samurai broke the deadlock in the 41st minute through Takuma Asano, before netting again in the 82nd thanks to Yosuke Ideguchi, locking in their ticket to Russia 2018 in the process.
Australia will now be looking for a big win against Thailand to ensure direct qualification for the Cup.
The opening half-hour was relatively tame, with Mathew Leckie coming close in the 38th minute when his shot from distance rebounded off Maya Yoshida.
Only moments later all their positive play was undone when Brad Smith left Takuma Asano unmarked in the box, with the Stuttgart forward converting the Yuto Nagatomo cross with ease, putting Japan ahead 1-0.
The Blue Samurai appeared to have the mental edge from then on, pressuring Australia's defence, with Hiroki Sakai and Yuya Asako all coming close, the Socceroos lucky to hold on.
That changed when Ideguchi rifled an unstoppable shot past Ryan and into the top corner from 20 yards out in the closing minutes, doubling the Japanese lead.
Now level on 16 points, but two goals behind Saudi Arabia, Australia will be hoping to pile the goals on against Thailand at home on Tuesday.
