The Socialist Party (PS) collected 36 percent in the second round of a poll to choose councillors in France's 100 departments, ahead of Sarkozy's UMP party with 20 percent and the far right National Front (FN) with 12 percent.
As in the first round turnout was low, with only 44 percent of those eligible to vote doing so, according to one polling firm.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
"The people of France have opened the way to change, and we are going to rush in," said Martine Aubry, leader of the PS and one of a number of possible Socialist challengers in the 2012 presidential election.
According to another leading Socialist Francois Hollande: "The lesson I draw from this vote is that the Nicolas Sarkozy page has been turned. The people of France want a new time, a new cycle."
The FN had aimed to pick up seats on several local councils but fell short of its hopes. Nonetheless it confirmed that is is a force to be reckoned with.
In the first round of voting last Sunday it registered 15 percent of the poll. It was the FN's best-ever performance at this level, reflecting a surge in interest in the party since Marine Le Pen, daughter of longtime leader Jean-Marie Le Pen, took the helm in January.
The new leader has tapped into French fear and resentment over the economic and social crisis.
Much to the discomfort of her political rivals, she has pushed the issue of Islam -- and the role of Muslims in French society -- squarely on to the national political agenda.
She presents herself as a plain-speaker -- but one who talks good, common sense.
"People will have to reckon with the FN coming in first place in the forthcoming elections, presidential and legislative," Marine Le Pen said.
"The redrawing of political life in France is under way," she said, claiming that she had succeeded in changing the FN vote from one of protest to one of membership.
An opinion poll published Sunday said Marine Le Pen would qualify for the second and decisive round of voting in next year's presidential poll whoever the Socialist candidate was and in most cases would knock out Sarkozy in the first round.
Conservatives suffered from the unpopularity of Sarkozy and kept a low profile after this latest setback, which followed defeats in European elections in 2009 and regional elections in 2010.
UMP leader Jean-François Cope said that the governing majority was "a little disappointed" but insisted that the PS was far from the results it had been counting on.
Prime Minister Francois Fillon said that"the left is making progress" but "the majority is dropping back less then announced."
He said that "in spite of a context made very difficult by two years of crisis, right wing and centrist candidates resisted well."
Fillon said that the outcome of the vote showed that "the protest vote should not be underestimated or seen as commonplace."
"This party has to be fought and the reasons for its following need to clearly evaluated and dealt with."
The PS seemed set to pick up several departments to add to the 58 it already controlled.
