The parliament failed to elect Speakers for either the lower house Chamber of Deputies or the upper house Senate.
It was the first meeting since Mr Prodi's Union coalition won a hotly-contested election earlier this month.
New votes for the speakers are expected at the weekend.
It was an inauspicious start for the 66-year-old former economics professor, as the secret ballots showed that many in his own disparate coalition voted against his nominees.
Mr Prodi had backed Franco Marini of the Margherita party, one of his coalition allies, for the Senate speaker's post.
He nominated Fausto Bertinotti, leader of the Refoundation Communist party, as leader of the lower house.
Marini failed to get elected in the first round, when at least one centre-left senator voted against him.
He appeared to have prevailed in a second round, reaching the required majority of 162.
But the acting speaker of the house, veteran former president Oscar Luigi Scalfaro, annulled the vote after three ballots were contested.
The votes bore a variation of Marini's name "Francesco Marini" and after protests from the centre-right bloc, Scalfaro ordered a repeat of the vote.
When the repeat came early Saturday he again fell one short of the 162 votes needed, as one ballot paper was declared void as his name was again spelled wrong.
A new vote was expected on Saturday at 0830 GMT.
No mistake:MP’s
Some senators questioned by AFP said the misspellings had not been an accident, but rather "a message" to Mr Prodi, who declined comment.
"They have tried three times, having on paper enough votes to get their candidate elected. They did not manage and showed that they are divided and without the structural majority to govern," the leader in the Senate of the right wing National Alliance Domenico Nania said.
The votes underlined the problems Mr Prodi, leading a disparate coalition with a tiny two-seat majority in the Senate, will face when he takes office next month.
Meanwhile, Mr Prodi's candidate for speaker of the lower house, Refoundation Communist leader Fausto Bertinotti, failed to reach the required two-thirds majority needed to gain the leadership of the Chamber of Deputies.
However, Bertinotti was expected to prevail at the latest in a fourth round, also likely on Saturday, when a simple majority is needed.
Mr Prodi's Union coalition holds 348 seats in the 630-seat lower house, with outgoing Prime Minister Silvio Berlusconi's centre-right coalition having 281.
However, in the 315-seat Senate the Union has 158 seats, just two more than Berlusconi's House of Freedoms.
His broad centre-left coalition will face immediate pressure when Mr Prodi seeks a vote of confidence in his government.
"Our objective is to bring it down as soon as possible," said outgoing Welfare Minister Roberto Maroni.
"The strategy is to make life impossible for the leftist government which is being formed," he said.
"Berlusconi has been very clear in his strategy. He reiterated it yesterday during the cabinet meeting," Maroni told the website Affaritaliani.
"Berlusconi knows that if it lasts five years, he will find it difficult to run as head of government again. But if it doesn't last even a year, he will go forward," said Maroni.
Berlusconi turns 70 in September.
The outgoing prime minister has indicated he will tender his resignation to President Carlo Azeglio Ciampi once the speakers of both houses are elected.