However both parties have acknowledged that wide differences remained, with Iran continuing to insist on its right to make nuclear fuel.
The West remains fearful that this could be used to manufacture nuclear weapons, a prospect that is keeping the two sides are far apart.
EU and Iranian officials said the five hours of talks were geared towards resuming formal negotiations that broke off in August.
The EU, during those negotiations, had offered trade and security incentives for Iran to abandon uranium enrichment.
Enrichment makes fuel for power reactors but also the raw material for nuclear weapons.
French foreign ministry political director Stanislas de Laboulaye told AFP that the Iranian and EU positions outlined on Wednesday in Vienna "are not the same. We repeated our positions and the Iranians repeated theirs."
An EU diplomat said negotiators from Britain, France and Germany, the so-called EU-3, warned the Iranians not to take any steps "between now and January" which are considered enrichment work, even if they fall short of actual enrichment.
Iran is reported to be considering taking such steps.
There should be no movement "in the manufacturing of centrifuge components and research on centrifuges," the machines that enrich uranium, the diplomat said.
Growing tension
The meeting came at a time of growing tension after Iran's President
Mahmoud Ahmadinejad raised an international outcry through a series of statements against Israel.
He enraged world leaders by his remark in October that the Jewish state should be wiped off the map.
The EU-3 are threatening to take Iran before the UN Security Council for possible sanctions but the new effort towards dialogue was greeted by one European diplomat as a sign that "at least it looks like the Iranians want negotiations."
"Both sides agreed to consult their respective leaderships with the view of holding another round of talks in January with the aim of agreeing on a framework for (formal) negotiations," De Laboulaye told reporters.
While the White House backed the EU-3's joint diplomatic efforts, one State Department official in Washington voiced exasperation about interrupting talks until January.
"Remember, the objective of this diplomatic exercise is not 'talk to talk,' it is negotiations to achieve an end. On that, everybody is unified," the official said, asking not to be named.
He said the Iranians "are going to have to overcome the presumption that they are not interested in negotiations. ... They have not give any indication to date that they are interested in resuming the talks in a serious manner."
The US claims that Iran is hiding the development of nuclear weapons under the cover of a civilian atomic program that Tehran says is peaceful.
Discussions heated
In Vienna, Iranian negotiator Javad Vaidi said the new talks would also be in the Austrian capital. He said he hoped the two sides would have "more opportunity" to move towards agreement.
Wednesday's talks were the first contact between the EU and Iran since August, when Iran resumed uranium conversion, thus torpedoing the EU-Iran negotiations.
Conversion is the first step in making enriched uranium.
Tehran claims it has the right to enrich under the nuclear Non-Proliferation Treaty, even if it is currently suspending enrichment as a confidence building measure.
Iran insists on the right to enrich uranium on its own soil, Iran's Foreign Minister Manuchehr Mottaki said in Tehran, apparently rejecting a Russian proposal that Iran to do some fuel work at home while enriching uranium only on Russian territory to keep this strategic activity out of Iran.
An EU diplomat said the divide between the West and Iran was so great that it "was unclear how there could be a compromise."
Mr De Laboulaye said the talks were "open and frank." A diplomat said this meant the discussion was "heated."
"The Iranians said they wanted to pursue their nuclear program. The
Europeans said they could do this in Russia but then the Iranians said foreign countries could do joint ventures in Iran in order to make sure that Iranian enrichment was not dangerous," the diplomat said.
"The real diplomatic work at the moment is trying to bring the Russians on board so we can take this to the Security Council," another EU-3 diplomat said.
But Russia, which has a veto on the Council, is building Iran's first nuclear reactor and says there is no sign Iran seeks atomic weapons.