In response to a series of threatening statements by Iran, including to erase Israel "from the map", Mr Olmert said: "We need to make sure no one has the capability or the power to commit destruction against us.
"The Jewish people and the very existence of Israel" are Iran's targets, Mr Olmert said. "We don't see enough of a response from the world."
In a satellite relay to a convention of the Anti-Defamation League, a pro-Israel group, he responded firmly to the Iranian leader's frequent rhetorical jabs at Israel.
"We do not take it lightly," Mr Olmert said, speaking from Jerusalem. "We are powerful and able to defend ourselves."
Olmert has urged the international community to work against Iran's nuclear program, saying Tehran's ambitions threaten not only Israel but all of Western civilisation. He called Mr Ahmadinejad a "dictator".
Growing concerns
Israel has long identified Iran as its biggest threat, and these concerns have grown amid repeated calls by Mr Ahmadinejad for Israel's destruction.
On top of the threats, the Iranian leader has denounced Israel as a "fake regime."
Mr Olmert's response came as Israel and Jews around the world commemorated the Holocaust, the attempt by the Nazis and their supporters in World War II to destroy European Jewry. An estimated six million Jews died in concentration camps and killing fields.
In his speech to the conference Mr Olmert reiterated his determination to establish a "new reality" in the region, either through negotiations or unilaterally.
He said Israel would separate itself from the Palestinians on parts of the West Bank, affording the Palestinians a state and a better life.
"We want to end this cycle of violence," Mr Olmert said, referring to the decades-old conflict between Arabs and Israel.
Mr Olmert also praised President Bush, saying Israel was "in good hands" when it relied on him.
The comments followed an announcement that Israel had launched a highly accurate imaging satellite that will enhance its ability to spy on Iran.