Needing a win to qualify for their first major tournament since the 2010 World Cup in South Africa, Serbia got the perfect start when Luka Milivojevic gave them an 11th-minute lead.
Austria, whose chances of qualifying had evaporated one hour before kick off when Wales won 1-0 in Georgia, hit back with goals from Guido Burgstaller and Marko Arnautovic before Nemanja Matic equalised for Serbia with seven minutes left.
But Louis Schaub struck in the 89th minute to give Austria a late victory.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
Serbia still lead the group with 18 points and will qualify if they beat Georgia at home on Monday. They are followed by Wales on 17 points and Ireland on 16.
"We are very disappointed but we weren't good enough tonight unlike Austria, who created a lot of chances," said Serbia coach Slavoljub Muslin.
"We have one more chance to reach Russia and we have to take it against Georgia at home. We can't dwell on what went wrong tonight."
Serbia had two early let-offs before Milivojevic put them ahead with their first attack, lashing a low shot into the bottom corner from outside the area after Maximilian Woeber's headed clearance landed at his feet.
The Serbian fans celebrated by letting off fireworks and a line of heavily armed police was sent in to stand guard in front of the visiting sector.
Austria levelled with the simplest of goals as Kevin Danso sent a long ball over a square Serbian defence into the path of Burgstaller, who was left with only Vladimir Stojkovic to beat and clipped the ball past the goalkeeper.
Austria took control after halftime, forcing Stojkovic to make a brave block at Arnautovic's feet and a superb save to tip away a Florian Grillitsch drive.
Arnautovic finished off a flowing move to put Austria ahead in the 76th minute, only for Matic to take advantage of hesitant defending and level seven minutes later.
Serbia sensed a possible winner - and qualification - but instead were caught out by Schaub's intended cross which bounced into the net after Arnautovic failed to make contact.
Arnautovic, who was born in Austria but has a Serbian father, did not celebrate his goal and told Serbian media afterwards he would support them against Georgia.
"I will be with Serbia on Monday. They are playing at home in front of a full house and I believe we will qualify for the World Cup," he said.
(Writing by Brian Homewood; additional reporting by Zoran Milosavljevic in Belgrade; Editing by Christian Radnedge and Toby Davis)
