Nic White came off the bench to score a penalty and a converted try which settled a contest which had see-sawed back and forth for the best part of 70 minutes, giving the Wallabies a first win over the All Blacks since they won the final Tri-Nations in 2011.
Prop Sekope Kepu and winger Adam Ashley-Cooper also crossed for the Australians, who will win back the Bledisloe Cup for the first time since 2002 if they can beat their neighbours in Auckland next week.
New Zealand winger Nehe Milner-Skudder scored two tries on debut and flyhalf Dan Carter kicked three penalties but it was not enough to prevent the All Blacks falling to only a third defeat since winning the World Cup in 2011.
"You don't just say you've won once in quite a long time and you've done something special," Australia coach Michael Cheika said, trying to dampen down expectations.
"It's nice for the lads, but it's an abridged Rugby Championship so it's only the three games. We want to test ourselves over more games."
All Blacks captain Richie McCaw suffered only his 15th defeat in the black shirt as he made his 141st appearance for his country, matching Irishman Brian O'Driscoll's record tally.
"I think Australia played exceptionally well tonight and thoroughly deserved the win," said All Blacks coach Steve Hansen.
"We're hurting. It still sucks even if it doesn't happen that often."
An uncharacteristic All Blacks spill from the kickoff and a good shove in the first scrum gave Australia early hope that it might be their night but they were soon defending their line after a burst from Milner-Skudder.
Kepu was sin-binned for killing the ball under his own posts and Carter slotted the ensuing penalty to give the visitors a 3-0 lead in the eighth minute.
David Pocock and Michael Hooper got the turnovers Australia coach Michael Cheika wanted from his experimental back row but poor handling and a shocking display of passing from scrumhalf Nick Phipps meant they were unable to exploit them.
Matt Giteau did put them on level terms in the 26th minute with his first penalty but Carter restored New Zealand's lead three minutes later.
Both sides would have been unhappy with their first half performances but the smiles were on the Australian faces in the crowd of 73,824 four minutes after the break.
SMITH CARDED
A high tackle on winger Adam Ashley-Cooper saw Aaron Smith yellow carded and Australia kicked for touch with the resultant penalty before driving to the New Zealand line.
When the rolling maul collapsed, Kepu crashed through the defence for a converted try which put the Wallabies 10-6 up.
Carter reduced the deficit with his third penalty before Phipps's evening went from bad to worse when he was shown a yellow card for dissent in the 54th minute. The All Blacks were back in front a minute later. Fullback Ben Smith somehow managed to get the ball away through a crowd of gold shirts to leave Milner-Skudder with a clear run to the line.
Carter missed the conversion and five minutes later Australia hit back when Matt Toomua's perfectly judged crosskick allowed Ashley-Cooper to touch down in the corner.
Giteau converted to give Australia a 17-14 lead but it lasted just three minutes before Milner-Skudder grabbed his second try, barging over the line through three tacklers and getting the ball down at the second attempt.
White put Australia a point ahead at 20-19 with a long range penalty after 69 minutes but better was to come from the replacement halfback and he cut through the visiting defence from close range three minutes later to secure the victory.
(Editing by Amlan Chakraborty)
