The turmoil threatened efforts by the Trump campaign to show the party had united behind the businessman-turned-politician and distracted from the day's theme of "Make America Safe Again," meant to depict Trump as a strong leader capable of shielding the country from violence and Islamist militancy.
The anti-Trump forces wanted to change the party's nominating rules to allow delegates to support alternative Republican candidates over Trump.
Party leaders held a voice vote, then declared the opponents lacked enough votes.
Watch the chaotic scenes from the Republican convention:
Pandemonium erupted on the floor of the Cleveland basketball arena where Trump is due to be formally nominated this week for the Nov. 8 election.
Many delegates began chanting "Roll Call. Roll Call," effectively calling for a lengthy process that would allow every state to weigh in.
Representative Steve Womack of Arkansas, who was chairing the session, declared there was not enough support for a roll-call vote, drawing a chorus of boos.
Some, including the Colorado delegation, walked off the convention floor saying they had to assess their next steps.
"This is really appalling," Ken Cuccinelli, a delegate from Virginia who favored a roll-call vote, told MSNBC afterward.
"This is the party of law and order. ... If you won't obey your own rules there is no reason to think you'll obey any others."
While delivering a sudden jolt to the highly scripted program, the anti-Trump forces failed, their rebellion quashed.
The convention then approved the party policy platform and took a scheduled break ahead of a lineup of evening speakers due to include Trump's wife Melania and former New York Mayor Rudy Giuliani.
But the furor, an embarrassment to Trump, put a spotlight on the deep divisions within the party that have emerged over his candidacy.
A string of senior Republicans, worried about Trump's temperament and policies, were already avoiding the convention.
Trump last week declared the so-called "Never Trump" movement dead, tweeting that "#NeverTrump is never more."
Killings overshadow convention
The gathering opened on Monday afternoon in the shadow of racially tinged killings of police officers and black men.
Trump allies planned to promote what he has billed as a tough line on law and order and national security in speeches later in the day.
Sunday's shooting of three policemen in Baton Rouge - a targeted attack that may have been in retaliation for a series of police killings of black Americans - hung over the gathering.
Trump has sought to position himself as the law-and-order candidate in an echo of Richard Nixon's successful presidential campaign of 1968.
Speakers were likely to promise that Trump will crack down on Islamic State abroad and toughen up on crime at home if he wins the election.
Iowa's Republican Party Chairman, Jeff Kaufmann, said that a month ago the top issue for voters in the state was the economy. Now, he said, he was hearing concerns about security.
"Rightly or wrongly, the shootings that we've had have vaulted, not just national security in terms of external terrorism but also the knowledge that that terrorism is occurring within our country," Kaufmann said.
Such concerns might lead voters to choose Trump over Democratic candidate Hillary Clinton in November, he said.
