The first Palestinian poll in a decade revealed increasing support for the Islamist group, a result that is likely to have a major impact on the Middle East peace process.
Fatah is expected to fall short of a majority, leading to the very real possibility that Hamas could demand a cabinet seat.
Both Israel and the United States have said they would not deal with a Palestinian government that includes Hamas.
Fatah is forecast to have won 63 seats in the 132-seat parliament, against 58 for Hamas, according to a poll by the West Bank's Bir Zeit university.
Fatah won 31 of its seats from voting on national lists while the other 32 were elected in constituencies, the poll said.
Hamas won 26 seats in the national lists and another 32 in constituencies.
The small leftist Popular Front for the Liberation of Palestine (PFLP) would win three seats while three other minor parties should win two seats each, the Bir Zeit pollsters said.
Another forecast by the Ramallah-based Palestinian Centre for Policy and Survey Research said Fatah won 42 percent of votes against 35 percent for Hamas, but did not give a breakdown of seats.
It also forecast that the PFLP had come third with five percent of votes.
The exit polls were released around an hour after all polling stations in the Gaza Strip, West Bank and annexed east Jerusalem were closed.
But Hamas has disputed Fatah’s lead as shown by the exit polls.
"The figures that we have show we have a big lead (over Fatah) and that the polls put out are inaccurate," Hamas spokesman Sami Abu Zurhi told news agency AFP.
"Hamas believes that its results are superior" to those in two officially-sanctioned exit polls, the spokesman added.
The election commission puts voter turnout at 73 percent.
Observers from the United Nations, the European Union and other international organisations said the poll passed off peacefully.
Historic poll
Palestinian leader Mahmoud Abbas called the election a decisive step on the road to independence.
After casting his ballot, he said the exercise in democracy should strengthen the prospects of peace and pledged his readiness to resume negotiations with Israel.
"The Israelis should have no reason to be fearful but rather pleased as we are building a democracy which can serve as a base for peace between us.
"After the elections I will form a new government and I am always ready for negotiations with the Israelis although they must want them on their side," Mr Abbas said.
Hamas, campaigning on a Change and Reform banner, has sought to cash in on disillusionment with Fatah over the stalled peace process, corruption and by claiming its fighters forced Israel to pull out of the Gaza Strip last summer.
"We want things to improve," said the elderly Abu Mohammed outside a Gaza City polling station. "I support Hamas. Why would I vote for Fatah? They live in ivory towers and leave the people with nothing."
Hamas’s leader in Gaza, Mahmud Zahar, said Hamas would be satisfied with an Israeli withdrawal from land captured in the 1967 Arab-Israeli war but only as a first step.
"We will first seek to establish a Palestinian state on any land we can recuperate, but without giving up any part of historic Palestine" in the long term, referring to the land that incorporates the state of Israel.
But Hamas's top candidate Ismail Haniya vowed that the militant movement would not disarm after entering parliament.
"The Americans and the Europeans say to Hamas: either you have weapons or you enter the legislative council. We say (we will have) weapons and the legislative council. There is no contradiction between the two."
’No to Hamas’
Israel’s acting Prime Minister Ehud Olmert has said Israel "cannot allow" the radical Islamist faction Hamas in its current form to become part of the Palestinian Authority.
Wednesday's poll was the first legislative election contested by Hamas, which has carried out most of the anti-Israeli attacks in the past five years and whose charter calls for the destruction of the Jewish state.
US hails poll
The White House on Wednesday hailed the "historic" Palestinian election but reiterated that it too would not deal with Hamas.
"This is a historic and significant moment for the Palestinian people. The Palestinian people are in a transition to what we all hope is a democratic state,” Spokesman Scott McClellan told reporters.
He refused to say whether Washington would cut aid to the Palestinians if Hamas joins any new government.
The US has previously warned the Palestinians that including Hamas in government could affect US-backed efforts to establish an independent Palestinian state.
The so-called Mideast Quartet – grouping the European Union, Russia, the UN and the US – will meet next Monday in London to evaluate the outcome of the Palestinian election.