"I believe I had the best prospect of leading the Liberal Party to success at the next election," he told reporters on Tuesday.
"That was not to be today and I understand and I respect the outcome and I fully support the Prime Minister and the Cabinet."
Mr Dutton has now resigned his powerful position as Home Affairs minister and will move to the backbench, in keeping with the tradition of cabinet solidarity.
“I made a decision not because I had any animosity towards Malcolm Turnbull … I made a decision to contest this ballot because I want to make sure we can keep Bill Shorten from ever being prime minister of this country."
Mr Dutton said it was good to be able to “smile” in front of the cameras and show a “different side” of himself, compared with his usual demeanour as the minister for Home Affairs.
He said he would now "do what I can as a backbencher to make sure that I support the government", but despite repeated questions from journalists he did not rule out launching another tilt at the leadership.
In a later interview with Sky News, Mr Dutton would not answer direct questions on whether he would challenge again.
"The arithmetic today is against me," he said.
"We’re on the day that the ballot’s just been had, so I think we respect the ballot and respect the decision that’s just been made."
Mr Dutton said the Coalition could win the next election if it focused on lowering power prices and reducing immigration until "infrastructure can catch up in our capital cities".
Turnbull calls for unity
Prime Minister Malcolm Turnbull called for unity and said he did not bear any "grudge" against Mr Dutton.
"We cannot allow, as I said in the party room today, our internal issues to undermine our work," Mr Turnbull told reporters on Tuesday afternoon.
"[It] creates a risk, a real risk, that Bill Shorten will be the prime minister."
Mr Turnbull confirmed he had "invited" Mr Dutton to remain in the cabinet as Home Affairs minister, but Mr Dutton declined.
Treasurer Scott Morrison will step in as acting minister, taking over the crucial portfolio in charge of Immigration, the federal police and Australia's lead intelligence agencies.
The government's chief whip Nola Marino confirmed the results of the leadership ballot in a courtyard at Parliament House on Tuesday morning.
The relatively narrow victory may not spell the end of Mr Dutton's leadership ambitions. Julia Gillard won her first ballot against Kevin Rudd 71 - 31 but went on to lose in a second ballot.
Only seven Liberals would need to change their vote to tip the balance in Mr Dutton's favour.
The contest came suddenly during a closed-door Liberal partyroom meeting when Mr Turnbull declared the leadership positions vacant, forcing challengers to declare their intentions.
Mr Dutton was the only challenger for the leadership. Julie Bishop was the only candidate for deputy and was re-elected without opposition.
Asked about the ballot, Ms Marino said the process was "orderly" as always.
Former prime minister Tony Abbott has led the charge as one of Mr Turnbull's main critics over the past few weeks, questioning the PM's "conviction" on Monday.
Mr Abbott said he told the party room “unity has to be created and loyalty has to be earned".
"They can’t just be demanded."
Nationals MP Darren Chester said it was time for the "circus to stop".
"The Australian people expect the circus to end now. They want us to back the prime minister we have got and carry on," Mr Chester said.
The ballot was just of Liberal members, who decide the leader of their party. It did not include the 22 members of the Nationals.
A second ballot?
Australian politics has a long tradition of leaders surviving initial challenges but losing in a second ballot.
Malcolm Turnbull himself survived a leadership spill motion when he was opposition leader, but was later defeated by Tony Abbott. Former Labor prime ministers Julia Gillard and Bob Hawke had similar experiences.
Mr Dutton previously said he would continue to support Mr Turnbull as long as he was a member of the cabinet. Now a backbencher, Mr Dutton is free to openly suggest policy alternatives.
Parliament sits for the rest of this week, then breaks until September 10.
Mr Dutton told Sky News there would be "no sniping" and "no undermining" from his position on the backbench, mirroring Tony Abbott's now-famous words when he lost the leadership to Mr Turnbull.
"And I can relax a little bit, which is good."
Labor attacks Coalition instability
The opposition launched a blistering attack on the prime minister in Question Time before calling for a vote of no confidence in his leadership.
"This house should vote for no confidence because the prime minister has no authority, no power, and no policies," Mr Shorten said.
"And the reason for that sits behind him. If nearly half of his own government do not want him to be the prime minister, why should the rest of Australia have to put up with them?"
The government, unusually, allowed the no-confidence motion to be debated.
The prime minister defended his mandate to govern, launching a counter-attack on Mr Shorten time as a union leader.
"The mandate our government has came from the 2016 election, remember that? We won and you lost. We are delivering."
The no-confidence motion was eventually defeated 67 - 76.
Turnbull's climate change reset fails to unite party
Earlier, Prime Minister Malcolm Turnbull tried to quell tension within party ranks by abandoning plans to legislate emissions reduction targets.
But the move has failed to silence conservative critics within the Coalition.
Former Prime Minister Tony Abbott said the backflip showed his successor was willing to abandon his core beliefs.
“What we want to know is, where are this prime minister's convictions?” Mr Abbott told reporters outside Parliament House on Monday evening.
“We always thought that he was convicted on climate change issues. I think he probably still is. And it was a conversion of convenience this morning.”