Second half goals from Tim Cahill and a sweetly struck long range shot from substitute Brent Holman put the Socceroos ahead, but with a poor goal difference from the 4-0 loss to Germany, more goals were needed.
A Ghanaian victory against the Germans - playing at the same time - would have meant Australian pasage to the second round with a vicotry over Serbia, but a Mesuet Oezil's beautifully struck strike gave the Germans victory.
The Socceroos hit back magnificently in the second half, rocking the Europeans with two goals in four minutes and shredding their highly-rated defence in their best performance of the tournament.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
It was to prove in vain, though, as Ghana, despite their 1-0 loss to Germany in Johannesburg, went through to the second round on goal difference.
Cahill heads Australia into contention
Everton midfelder Tim Cahill started it off for the Australians by outjumping Manchester United central defender Nemanja Vidic and heading home in the 69th minute before substitute Brett Holman thumped in a long-range shot four minutes later.
Substitute Marko Pantelic pulled one back with six minutes left, had another chalked off for offside and then put another one over the bar with the goal -- and a place in the last 16 -- at his mercy.
"I have mixed feelings now," Australia coach Pim Verbeek said.
"It was a great result and a great performance but we are not in the final 16.
"That was the plan when we came to South Africa, but it didn't work out that way and I cannot blame any one of the players, who worked very hard last week when everybody thought it was over.
"Four points was enough for Australia in Germany four years ago, but not this time."
Serbia were favourites to advance
Serbia had been favourites to advance after beating Germany 1-0, while Australia ended their ill-fated campaign on a winning note after being routed 4-0 by the Germans and drawing 1-1 with 10 men against Ghana.
"This was a game of good quality. I cannot reproach any of my players for their effort," Serbian coach Radomir Antic said.
"We had plenty of opportunities but in the end I think that we didn't get fair treatment. We deserved much more than we got. We have to go back home now."
Serbia looked the better team in a scoreless opening half, dominating possession and territory.
Right-winger Milos Krasic looked particularly dangerous and had two scoring chances but Australia saw off the threat.
Australia goalkeeper Mark Schwarzer made a great close-range block of Branislav Ivanovic's shot on 23 minutes after a David Carney error inside the area.
Australia's attack was well corralled by Vidic, with the Serb back four winning virtually all the aerial duels.
Krasic had the ball in the net on 38 minutes but was flagged offside.
Australia picked up two yellow cards with defenders Michael Beauchamp and Luke Wilkshire going into the referee's book early in the second half.
Antic took off Krasic just after the hour and brought on Zoran Tosic as Australia began to have more of the play.
Verbeek made a double substitution by throwing on attackers Scott Chipperfield and Brett Holman on 66 minutes and it was to pay swift dividends.
Brett Emerton became the third Australian booked for a 50-50 challenge on Milos Ninkovic as Antic took off giant striker Nikola Zigic and brought on Pantelic.
Australia's persistence finally paid off on 69 minutes when Cahill outjumped Vidic from Wilkshire's cross and headed home past the despairing Vladimir Stojkovic.
Holman let fly with a low, long-range strike four minutes later to put Australia 2-0 ahead to a roar from the huge Australian support.
But Pantelic grabbed one back after Schwarzer could not cleanly claim Tosic's volley, before fluffing his lines by shooting over in injury time with the goal gaping.
