Simon Msuva and Mbwana Samatta put Tanzania in front but Olunga and Johanna Omolo levelled for the Harambee Stars before Olunga struck the winner nine minutes from time.
Kenya, who had won only once in 15 previous matches in the competition, have three points from two games in Group C, level with Senegal. Tanzania, yet to get off the mark, became the first team to be eliminated in the 24-team tournament.
"It was completely crazy and the coach at this time is just a spectator," Kenya coach Sebastien Migne said. "The players wrote history tonight, now everything is possible, we have nothing to lose."
Tanzania coach Emmanuel Amunike was surprisingly critical of his team.
"There were a lot of crazy things....we committed a lot of mistakes," said the former Nigeria international. "You practise and practise, but players are players and as a coach you stand on the touch-line and you are helpless."
Although neither side are among the heavyweights of African football, they produced a remarkably open contest with a hatful of chances.
Migne said before the match that it was time for his team, who are taking part at their sixth Afcon tournament, to end their run of falling at the first hurdle each time.
But they were caught napping in the sixth minute as Tanzania took a shock lead. An end to end move ended with Samatta being sent clear down the left, his shot was parried by Patrick Matasi and Msuva turned in the rebound.
Helped by the generous amount of space given to them by Tanzania, Kenya began to create openings and Erick Ouma had a close-range effort brilliantly stopped by goalkeeper Aishi Manula before Olunga hit the crossbar
They equalised in the 39th minute with an acrobatic effort. A free kick into the area was pushed away by Manula, the ball bounced off Tanzania defender Erasto Nyoni and was hooked into the net by Olunga who showed excellent reactions.
Kenya still seemed to be celebrating one minute later when Samatta won the ball in the penalty area and the Belgian-based forward fired a low shot into the net.
The game continued in the same vein in the second half and Kenya levelled again in the 63rd minute, Omolo scoring with a diving header from a low cross at the near post.
Kenya snatched victory when they broke out of defence and the ball was played to Olunga, whose snot sneaked between Manula and his near post.
(Reporting by Brian Homewood, editing by Ed Osmond)
