Triple champion Hamilton, 26 points behind Rosberg with three races remaining, had set the pace on a chilly and overcast morning at the Autodromo Hermanos Rodriguez with a lap of one minute 20.914 seconds.
Vettel ousted the Briton from the top in the afternoon, by just 0.004 of a second, with a lap of 1:19.790 on supersoft tyres that showed Ferrari are not about to let Sunday's race turn into another Mercedes duel.
It was the first time the Italian team had led a Formula One practice session since Belgium in August and Vettel was pumped up, calling McLaren's Fernando Alonso 'an idiot' after being slowed by the Spaniard.
"Maybe it was just me today, but I felt that I was actually in Mexico City, not on the race track, with a lot of traffic," said the four times champion.
"I think overall the pace was good, which was the more important thing. We can still improve for tomorrow, and then go from there."
A more competitive Ferrari could be good news for Hamilton if he can take the victory and have others finishing between him and Rosberg.
Rosberg, winner of nine races to Hamilton's seven so far this season, was seventh on the morning timesheets and 0.759 off Hamilton's pace but moved up to third in the afternoon -- albeit still 0.431 slower than his team mate.
WORK TO DO
Mercedes said the championship leader had been running through his morning programme as planned and had not suffered any problem, but Rosberg said he had work to do.
"It's been a bit of a steep learning curve to understand what to do with the tyres, how to get the best out of them for one lap and also for the long runs," said the German.
"We really learned a lot today and it's not been a bad start overall but we have a lot of homework to do tonight...Ferrari seemed very quick here today so they will be a big threat."
Hamilton, who picked up where he left off after a dominant win in Texas last weekend, needs to win on Sunday to boost his flagging hopes of a fourth title while Rosberg would be crowned champion if he wins and his team mate finishes 10th or lower.
The Briton said the track felt a lot better than last year, when it returned after a 23-year absence, and he felt more comfortable in the car.
Mercedes have already clinched the constructors' championship for the third year in a row while Rosberg and Hamilton are the only drivers in contention for the other title.
Vettel's team mate Kimi Raikkonen, who was third in the morning, was fourth after lunch.
Mexican Sergio Perez kept the home crowd cheering with fourth place ahead of Force India team mate Nico Hulkenberg in the opening session but dropped back to 15th in the afternoon.
The morning session was red-flagged when Sauber's Brazilian Felipe Nasr ran bumpily over a kerb, with the right section of his car's front wing shattering in a cloud of carbon fibre debris.
Red Bull's Daniel Ricciardo, third in Texas behind the Mercedes pair, was eighth and fifth fastest respectively in the sessions with Dutch team mate Max Verstappen 14th and seventh.
Verstappen had a moment of early drama when mechanics had to use fire extinguishers on his flaming rear brakes.
(Reporting by Alan Baldwin, editing by Toby Davis)
