Palace held on for their first away win over Newcastle in nine attempts, a run stretching back to 1998.
It lifted Roy Hodgson's side to 12th with 39 points, 11 ahead of third-bottom Cardiff City. Newcastle remained in danger, though, dropping to 15th with 35 points.
To make matters worse for the hosts they had Florian Lejeune taken off on a stretcher in the second half with a knee injury.
Defeat ended a run of five successive home league wins for Rafa Benitez's side.
Palace played on the counter-attack throughout with Andros Townsend and Zaha a constant threat.
With 10 minutes remaining and Newcastle seeking to extend their impressive home run, another rapid Palace breakaway left the hosts exposed and Zaha skipped past Fabian Schar before being fouled by Yedlin.
Milivojevic has now scored 11 goals in all competitions this season, 10 of them from the penalty spot.
"This three points will guarantee (us staying up)," he said.
"Today our game over 90 minutes was very good. We controlled the game in the second half."
Both sides had goals ruled out in a lively first half with Salomon Rondon denied for the hosts and James Tomkins, who also hobbled off late on, for Palace.
Newcastle will be scratching their heads over their failure to pick up at least a point after having the lion's share of possession and chances. They remain winless in 11 Premier League matches against London clubs this season.
"It's a big disappointment because we had a lot of chances and lots of shots on goal but this is sometimes the kind of game you cannot win but it is important not to lose," Benitez said.
"The most difficult one is the final third. We pushed and we attacked but we didn't control the counter-attack the way we should have done."
(Reporting by Martyn Herman, editing by Ed Osmond)
