The visitors scored two tries -- one a penalty try and the other from prop WP Nel -- and were reliant on the unerring goalkicking boot of Greig Laidlaw, who made six out of six for 16 points.
The injury-hit home side showed bursts of attacking flair but were left to rue a series of disciplinary lapses, which at one stage either side of halftime left them with 13 men after two players were sent to the sin-bin.
The Scots had handed Japan their only loss in the pool stage of last year's World Cup, leaving the hosts thirsting for revenge, and they started their mission promisingly.
Threading together a succession of passes in the ninth minute, hooker and captain Shota Horie ploughed over, the converted try answering Laidlaw's early penalty.
Yet Japan's indiscipline enabled the visitors to bounce back, allowing the Scotland captain Laidlaw to punish the hosts with his boot.
Flanker Hendrik Tui was shown a yellow card in the 35th minute and fullback Rikiya Matsuda, in for the injured Kotaro Matsushima, was also sent to the bin three minutes later for an intentional knock-on.
For Matsuda's infringement, referee Ben O'Keeffe awarded Scotland a penalty try, which Laidlaw converted to give his team a 16-10 half time lead.
Scotland capitalised on their two-man advantage by adding a converted try soon after the break through Nel.
Laidlaw stretched their lead to a commanding 26-13 with another penalty in the 62nd minute.
Japan had to manage without some of their World Cup stalwarts like scrum-half Fumiaki Tanaka, loose forward Michael Leitch and fullback Ayumu Goromaru, all under contract with Super Rugby teams this season.
"We gave away too many penalties. That's an area we will work to fix before we challenge Scotland again," Horie said after the match.
The Brave Blossoms will have to regroup for the deciding test at Tokyo's Ajinomoto Stadium next Saturday.
(Reporting by Shinichi Saoshiro; Editing by Ian Chadband)
