The early results suggest Anies Baswedan has won with 58 per cent of the vote against 42 per cent for Jakarta's standing governor, Basuki Tjahaja Purnama, a Christian.
Anies Baswedan's huge victory margin has come as a shock since opinion polls in the run-up to the election had pointed to a virtual tie.
Basuki Tjahaja Purnama, better known as Ahok, won the first round of voting for governor in February in a three-way race.
But despite the disappointment, he has urged his supporters not to be sad and wished Mr Baswedan and his running mate, Sandiaga Uno, well.
"We hope Anies and Sandi rule Jakarta well. And to our supporters, we understand you must be upset, but remember this: Power is given and taken away by God. No-one can control that."
Mr Baswedan came under criticism in the election campaign when he aggressively courted the Islamic vote, appearing publicly with controversial leaders at anti-Purnama rallies.
His platform has focused on improving public education, providing no-deposit home loans for low-income groups and opposing a giant seawall in Jakarta Bay that Ahok backed.
Mr Baswedan denied he planned to implement Islamic sharia law if elected.
He has promised to keep fighting for equality in Jakarta.
"Our consistent commitment is to focus on social justice, which means we aim to end the problems that plague Jakarta now, like inequality. Jakarta is the most diverse city in Indonesia, and our commitment is to safeguard the diversity and fight for the unity of our people."
Sandiaga Uno has urged all residents to focus on the future.
"We are friends, a united Jakarta. Forget the last few months, and let's face the next five years, full of glory."
The election drew widespread criticism from both sides, with Ahok on trial for blasphemy over comments he made last year that many took to be insulting to Islam.
Hundreds of thousands of Muslims took to the streets in response to calls for his sacking and urging voters not to elect a non-Muslim leader.
One person died, and more than a hundred were injured as a result.
An analyst at the Jakarta-based SMRC research centre, Ade Armando, says he has concerns now about future elections and former presidential hopeful Prabowo Subianto.
"This is like breathing a new life, a new strength, for Prabowo's faction to be able to return. What we are worried about is, now, they think using religion is a successful way of taking down the oppositions and they'll keep doing it."
Residents in Jakarta have shown a mix of reactions to the results.
Street vendor Farid Alfarizi says he is pleased with the outcome.
"I'm happy, very happy, because Muslims are obligated to elect a Muslim leader. Secondly, as a street vendor, I have suffered from the raids enforced by the city council officials, so maybe, if Anies and Sandi are elected, our life will be easier."
But local resident Neneng Herbawati says Jakarta residents voted irrationally for the man he called Candidate Number 3.
"Because of religion, they dismissed the track records, and they also didn't see Candidate Number 3 (Anies Baswedan) based on his personality but based on religion."
Mr Baswedan will officially take over as governor in October.
Share
