The Serb rose to power a freekick from Frank Lampard into the back of the net on 73 minutes, after Christian Benteke had cancelled out an early own goal from Villa defender Antonio Luna.
The Chelsea defender was perhaps lucky to be on the pitch, however, as moments before he grabbed the winner he was adjudged to have elbowed Benteke with referee Kevin Friend producing only a yellow card when it could have been red.
The Villa dugout was up in arms again in stoppage time when John Terry appeared to handle a cross from Fabian Delph but Friend waved away their appeals.
The win means Chelsea have a perfect six points from their opening two league matches after beating Hull City 2-0 on Sunday, while Villa could not repeat their shock win 3-1 over Arsenal on Saturday.
After a patient and largely pedestrian start, Chelsea were handed the opener on a plate.
Oscar set Eden Hazard through down the left and the Belgian's side-footed finish was parried by Brad Guzan into the path of Luna, who, oblivious to the danger, bundled it into his own net.
The hosts, however, failed to capitalise on their advantage and Benteke ensured they paid the price with the leveller on the stroke of halftime.
Gabriel Agbonlahor followed an injection of pace down the left with a low cross that found Benteke in space. He switched it from his right foot to his left and blasted home from 10 metres out.
Villa had great chances to go ahead in the second half with Agbonlahor sidefooting a good chance over and Chelsea keeper Petr Cech producing two great saves to deny Andreas Weimann either side of Ivanovic's winner.
There was further controversy in stoppage time when Villa were denied a clear penalty as Terry jumped to clear a cross from Delph, but caught the ball with his flailing arm.
(Reporting by Toby Davis; Editing by Sonia Oxley)
