It sets up an intriguing final day with Smith's team needing 185 runs to go 1-0 up in the three-test series while Sri Lanka require seven wickets to clinch the bowler-dominated contest.
Earlier in the morning, Kusal Mendis fell for a marathon 176 but the Sri Lankan tail wagged as the hosts, resuming on 282 for six, posted 353 all out in their second innings.
Mendis added seven runs to his overnight before Mitchell Starc (4-84) dismissed him caught behind.
Rangana Herath (35) frustrated the Australians with his belligerent batting until substitute fielder Moises Henriques took a stunning one-handed diving catch to send him back.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
Herath was soon back to torment the touring side again, this time with the ball, when the left-arm spinner bowled David Warner who ended the match with scores of one and zero.
Off-spinner Dilruwan Perera trapped Usman Khawaja (18) lbw and left-arm Chinaman bowler Lakshan Sandakan bowled Joe Burns (29) with a sharp-turning delivery to reduce Australia to 63-3.
Australia looked in further trouble when Voges was adjudged leg-before for a golden duck after being struck on his pad by a Perera delivery.
The batsman reviewed the decision and got it overturned after replays suggested the sharp-turning ball would have missed leg stump.
Australia are without their hamstrung left-arm spinner Steve O'Keefe, who hobbled off the ground on Thursday and is set to miss the rest of the series.
(Reporting by Amlan Chakraborty in New Delhi; Editing by John O'Brien and Ed Osmond)
