Labor is likely to lose at least three seats in Victoria and has failed to regain the prized seat of Melbourne from the Greens.
The Liberal Party is well ahead in the seats of Corangamite, Deakin and La Trobe with significant portions of the vote counted.
In McEwen, Labor leads the two-party preferred vote by a little over one per cent.
Labor had hoped it would gain the seat of Melbourne, but Greens MP Adam Bandt has retained the seat with a two-party preferred vote of nearly 56 per cent.
Mr Bandt said it was a win for the people and the refugees.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
"There's lots of people who think that there's a better way," Mr Bandt said.
"The real opposition and the real alternative to Tony Abbott's agenda is going to be led by the people of Melbourne in the House of Representatives."
There were significant swings recorded against Labor in several other Labor seats including Bendigo and Ballarat, but the ALP retained the seats.
Former Speaker Anna Burke looks to have held on to her seat of Chisholm, leading by about four per cent on the two-party preferred vote.
In the country Victorian seat of Indi, Liberal frontbencher Sophie Mirabella is facing a tough challenge to retain her seat.
With 50 per cent of the vote counted Ms Mirabella had won 43.4 per cent of the vote compared with independent Cathy McGowan's 32.9 per cent.
Labor has recorded about 11.2 per cent of the vote and the Greens about 3.7 per cent.
The majority of preferences from Labor and the Greens are expected to flow to Ms McGowan.
In the seat of Mallee, the Nationals were battling it out with their coalition counterparts the Liberals, but it appears the Nationals have retained the seat.
The Nationals recorded more than 41.2 per cent of the primary vote, compared to the Liberals' 24.65 per cent.
Victoria's Liberal Premier Denis Napthine said Mr Abbott would be an outstanding prime minister.
"Tony Abbott and his team will better manage the economy, boost confidence and jobs and create greater opportunities for Victorian businesses and families," Dr Napthine said in a statement.
Mr Abbott has pledged $1.5 billion from a coalition government for Melbourne's East West Link road project.
"With a strong swing against Labor in Melbourne's east, this election result demonstrates overwhelming support for the East West Link project," Dr Napthine said.
