Police persuaded three settler families, including women and children, to leave but had to drag or carry out two dozen teenage supporters holed up inside the house, near a heavily fortified Jewish settlement in the biblical city.
Security forces had scuffled with scores of settlers outside after some threw petrol bombs from the roof of the three-storey building and others hurled rocks. Police arrested 19 settlers, while 17 policemen were lightly injured.
"We will respond immediately to any case where there is a violation of the law and any attempt to determine facts on the ground," Mr Olmert told the first meeting of his new cabinet.
The house was not within the enclave of Hebron that Israel recognises as a Jewish settlement, and its evacuation offered a taste of what may happen if Mr Olmert implements his plan to impose final borders for Israel by 2010.
In the absence of peace talks with the Palestinians, the plan calls for dozens of isolated Jewish settlements in the occupied West Bank to be evacuated, while major blocks are retained and expanded behind a fortified border.
"In the next few years, we will change Israel's character to ensure it will be a country with a solid Jewish majority living in defensible borders that can provide security for residents of Israel and separate us from those who live alongside us and not among us," Mr Olmert said at a ceremony marking his move into the prime minister's office.
Mr Olmert added settler violence would not be tolerated.
"We shall not accept thuggery and attempts to illegally establish facts on the ground," he said. He also made clear spending on illegal Jewish settlements will be vastly reduced and will instead be concentrated on relatively underdeveloped areas such as the southern Negev and northern Galilee regions.
Mr Olmert took over as interim prime minister in January after Ariel Sharon was incapacitated by a stroke, but he had not worked out of the prime minister's office until the weekend, three days after his coalition government was sworn in.
Deadline for peace talks
Jordan's King Abdullah said Mr Olmert's plan set an effective deadline for currently stalled peace efforts.
"The time available to us for a peaceful settlement is around two years and I fear, if this short time is over and we don't reach a settlement, there will be nothing left for the Palestinians to negotiate over," he told al-Arabiya television.
The settlers moved into the house last month, saying it had been bought from its Palestinian owners legally. Palestinian groups denied this, suspecting a gradual attempt to expand the settlement, despised as a symbol and reminder of occupation.
Israel's Supreme Court on Thursday ordered the settlers to be evicted pending a ruling on ownership. Police said 41 settlers had been inside the house, refusing to leave. The police used an electric saw and a sledgehammer to break in.
Some of the women from the families wept as they walked out. "Palestinians will receive much strength today. There is no justice and no righteousness in this corrupt state," said Tzippora Schlissel, 40.
But there was relief among Palestinian neighbours. "The last month has been very difficult for us. We have had stones thrown at us and our electricity and water tampered with. I wish all the settlers in Hebron would leave," said Umm Nemer, a 43-year-old mother of eight.
Mr Olmert has not said what his plan would mean for the 400 settlers in Hebron, who live among 130,000 Palestinians. The city, a flashpoint of the conflict, is holy to both Jews and Muslims as the burial place of biblical patriarchs.
Last year Israel evacuated 9,000 Jewish settlers from the Gaza Strip and a remote northern part of the West Bank, amid widespread resistance.
In February about 200 protesters and police were injured in clashes when Israel dismantled part of an unauthorised settler outpost in the West Bank. Police said it was the fiercest violence they had ever faced from Israeli Jews.
About 240,000 settlers live in the West Bank among 2.4 million Palestinians and Mr Olmert's plan would involve moving about a quarter of them.
The Palestinians reject his plan out of hand, saying it means an annexation of their land that will permanently deny them the viable state they want in the West Bank and Gaza Strip.
The World Court has branded all Israeli settlements built on land captured in the 1967 Middle East war as illegal. Israel disputes this.
Buffett invests
In a significant boost to Mr Olmert's economic policy, the US investor Warren Buffett closed a deal over the weekend to acquire the Galilee-based metalworks company Iscar for US$4 billion - giving the Israeli government a tax windfall of around US$1 billion.
"This is not merely an investment of billions of dollars. This is the largest investor on the face of the earth," Mr Olmert told reporters on Sunday.
"He (Buffet) is not Jewish, nor is he a Zionist. The Israeli economy is such that he believes in it and supports it," he said.