An absolute majority of 180 lawmakers voted for the motion to loud applause and shouts of "Yes we can," converting Rajoy into the first prime minister to be ousted by such a vote since Spain transitioned to democracy in 1977.
The bespectacled 63-year-old leader got up and shook hands with Sanchez before leaving the lower house without a word.
Rajoy had already admitted defeat minutes before the vote, knowing that an absolute majority of lawmakers as diverse as Catalan separatists and Basque nationalists had pledged their support for the no-confidence motion.
"It's been an honour - there is none bigger - to have been Spain's prime minister," he told parliament, with lawmakers from his conservative Popular Party (PP) giving him a standing ovation.
Spanish Prime Minister Mariano Rajoy applauded by his fellow party members. Source: AAP
Sanchez, Spain's 46-year-old opposition leader, had instigated the no-confidence motion last week after a court revealed details of a vast system of bribes given to former PP officials in exchange for lucrative public contracts between 1999-2005.
After years of anger over the scandals tainting the PP, corruption finally got the better of the party and sealed Rajoy's downfall.
"Today we are signing a new page in the history of democracy in our country," Sanchez told parliament prior to the vote.
But PP lawmaker Rafael Hernando told him he would be entering the prime minister's office "through the back door" after failing to win 2015 and 2016 general elections.
"For the first time we may get a prime minister who didn't win elections," he retorted.
Socialist leader Pedro Sanchez in the parliament after the motion of no confidence vote. Source: AAP
Tough ride for Sanchez
In order to push through the no-confidence motion, the Socialists, who hold just 84 of the parliament's 350 seats, had to cosy up to parties they had previously clashed with, like Catalan separatists and the anti-establishment Podemos.
As such, even if he has pledged to govern long enough to restore "institutional stability" before calling early elections, Sanchez's new government will likely be highly unstable.
Podemos has already asked to be part of his new government.
Aitor Esteban of the Basque PNV nationalist party, whose support proved decisive for the motion's success, on Thursday warned that such a minority government would be "weak and difficult, complicated."
"This is going to be a constant bing, bang, boom."
A scandal too far
Although Rajoy survived a similar no-confidence vote last year, Friday's ballot draws a line under his rollercoaster time in office which began in 2011 and saw him implementing drastic spending cuts before winning re-election in 2015 and 2016.
"Today we are signing a new page in the history of democracy in our country." Sanchez
Despite winning the last two votes, he lacked the absolute majority of his first term.
He put Spain back onto the path of growth after a devastating economic crisis although unemployment remains sky-high, jobs precarious and many complain inequalities have risen.
But his term in office was also marred by a series of corruption scandals involving former PP members.
Last week the National Court, which deals with major criminal cases, sentenced 29 people with links to the PP, including a former treasurer, to a total of 351 years in jail.
It also ordered the party to pay back 245,000 euros ($290,000) received from the scheme to help finance election campaigns.
'What moral authority'?
Rajoy became Spain's first sitting prime minister to give evidence at trial when he was called as a witness last year.
In its ruling, the court said the credibility of Rajoy's testimony "should be questioned".
During Thursday's pre-vote debate, Rajoy said the corruption case "does not concern members of the government" and repeated the party's argument that only a tiny number of its politicians have been tainted by corruption.
He also hit back by listing the many graft cases involving the Socialists over the years.
"Are you Mother Teresa of Calcutta? With what moral authority do you speak?" he told Sanchez.
