The security cabinet accepted "the recommendation of all the security bodies to change the inspection with metal detectors to a security inspection based on advanced technologies and other means," a statement from Prime Minister Benjamin Netanyahu's office said.
As word spread of the decision, a few hundred Palestinians gathered to celebrate near an entrance to the Haram al-Sharif mosque compound, known to Jews as the Temple Mount.
One person set off a firework, prompting Israeli police to raid and disperse them using sound grenades.
Israel installed metal detectors at entrances to the Jerusalem site, which includes the Al-Aqsa mosque and the Dome of the Rock, after an attack on July 14 that killed two policemen.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
Palestinians view the new security measures as Israel asserting further control over the site. They have refused to enter the compound in protest and have prayed in the streets outside instead.
Israeli authorities said the metal detectors were needed because the July 14 attackers smuggled guns into the site and emerged from it to shoot the officers.
Clashes have broken out during protests over the measures, leaving five Palestinians dead.
Three Israelis were also killed when a Palestinian sneaked into a house in a West Bank settlement and stabbed them.
The decision to remove the metal detectors follows talks between Netanyahu and Jordan's King Abdullah II.
Jordan is the official custodian of Muslim holy sites in Jerusalem.
