When he takes power, Mahathir will be the oldest prime minister in the world.
Despite the shock result there were no reports of trouble on the streets, where pockets of celebration erupted overnight.
A flag-waving crowd of supporters gathered on a field outside the headquarters of Mahathir's party near Kuala Lumpur.
Suva Selvan, a 48-year-old doctor, said he felt the country had just won its independence.
"I feel that with this change we probably can see something better in the future... our hope for the future is a better government, fair, free and united," he told AFP.
Defeat could just be the beginning of Najib's troubles. Mahathir has vowed to bring him to justice over allegations that billions of dollars were looted from sovereign wealth fund 1MDB, which the scandal-hit leader set up and oversaw.
But at a press conference after his win, Mahathir vowed: "We are not seeking revenge. We want to restore the rule of law."
He said he would be sworn in Thursday.
Mahathir's return to the political frontlines saw him throw in his lot with an opposition alliance filled with parties that he crushed while in power, and which includes jailed opposition icon Anwar Ibrahim -- his former nemesis.
Anwar is due to be released from prison in June. Mahathir has vowed to get him a royal pardon, and later hand over the premiership to him.
As well as seizing control of the national government, several state legislatures across the country fell into opposition hands for the first time, including the highly symbolic bastion of Johor, the birthplace of Najib's party that was the lynchpin of the ruling coalition.
Official results from the Election Commission showed that Mahathir's opposition grouping Pakatan Harapan, along with a small ally, had secured 121 parliamentary seats. 112 are needed to form a government.
BN got 79 seats –- a dramatic drop from the 133 they held previously.
Najib was nowhere to be seen as the evening wore on, and was believed to have holed up inside his house for talks with senior members of BN.
The opposition win was particularly striking as critics said that Najib tried everything he could to cling to power.
His government was accused of pushing a redrawing of electoral boundaries through parliament that created constituencies packed with the country's Muslim Malay majority, who have traditionally supported BN.
He called the election on a weekday -- polls are typically held at the weekend in Malaysia -- in what observers said was an effort to keep turnout down, which would hurt the opposition. In the event, voters flocked to the polls in huge numbers.
Ultimately the explosive allegations of corruption, coupled with anger at rising living costs, proved too much for Malaysia's 15 million voters, already sick of racially divisive politics in the multi-ethnic country and graft scandals under years of BN rule.
In Mahathir, the opposition found the perfect person to take on Najib. He is a staunch Malay nationalist who could appeal to the country's biggest ethnic group, and whose years in power were remembered as a prosperous period in the country's history.
The initial euphoria at the opposition victory will likely give way to some apprehension.
Mahathir was also a accused of being an authoritarian leader, and political opponents were thrown in jail during his time in office.
Australian Prime Minister Malcolm Turnbull said he was looking forward to working with the Malaysian government following the election results.
"We will wait and see. It is the decision of the Malaysian people and we have always worked well with the government of Malaysia and we look forward to the results of the election," he told ABC Radio.