President Abbas has denied demanding a prisoner release as a condition for a summit.
Israel is insisting that before it frees any of the estimated 8,000 Palestinians it is holding, a soldier captured in June by Hamas-linked militants in a cross-border raid must be returned.
Mr Olmert announced late last month he hoped to soon hold a meeting with Mr Abbas, but preparations became bogged down over the prisoners.
The two have met only once since Mr Olmert took office in March, during an informal chat in Jordan.
"We offered to meet with Abu Mazen (Mr Abbas) but apparently he is not interested," Mr Olmert was quoted as telling a meeting of lawmakers from his Kadima party. "He is conditioning a meeting on the release of prisoners and we will not release any prisoners until (captured soldier) Gilad Shalit is released."
The official, who attended the meeting, spoke on condition of anonymity because he was not authorised to discuss the matter with reporters.
But the disagreements over prisoners and a summit are not the main impediments to diplomatic progress.
Internal Palestinian and Israeli political turmoil and control of the Palestinian government by of the extremist Islamic Hamas, which does not recognise Israel or agree to renounce violence are larger obstacles in the way of US efforts toward peace talks, which have been stalled for six years.
US Secretary of State Condoleezza Rice was in the region this month to try to prop up Mr Abbas, leader of the moderate Fatah Party, which was ousted by Hamas in January elections. Mr Abbas was elected separately a year earlier and remains in office, but his power and influence have been severely curtailed.
Also, Mr Abbas has been unable to persuade Hamas to moderate its positions to allow his party to come back into the government - the only realistic possibility for persuading the West to end its punishing aid embargo against the Hamas regime that has bankrupted the Palestinian Authority, causing widespread hardships that have spilled over into Fatah-Hamas clashes.
Mr Olmert is embroiled in his own political problems, his popular support decimated by the summer's war against Hezbollah in Lebanon that produced questionable results.
His solution is to woo an extreme hardline party to shore up his coalition majority, angering the dovish members of his current team and further clouding peace prospects with the Palestinians.
In the absence of negotiations, violence has captured centre stage.
Israel has been carrying out a widespread offensive in Gaza since the soldier was captured. Israeli forces have killed 21 Palestinians in fighting since last Thursday. Most of the dead were militants, but two children, ages 10 and 14, and a 29-year-old woman also were killed.
Israel says the latest raid is aimed at targeting the repeated fire of homemade rockets from the northern Gaza Strip.