Politicians in the devolved Edinburgh assembly voted by 93 to 30 to refuse "legislative consent" for the highly-contested European Union (Withdrawal) Bill currently being debated by the British parliament.
British Prime Minister Theresa May is under no obligation to amend her Brexit plan in response to Holyrood's objection.
However, experts warn that confrontation between London and Edinburgh could push Scotland towards independence.
Scottish First Minister Nicola Sturgeon, leader of the secessionist Scottish National Party (SNP), said Britain was heading into "uncharted constitutional territory".
The dispute centres on who will have control of powers currently residing in Brussels, such as over farming and fisheries, once Britain leaves the EU.
Scotland's SNP government wants such powers to be under Scottish control, while the Conservative British government argues they should reside in London, at least initially.
Michael Russell, the Scottish government's chief Brexit negotiator, said any attempt to use Brexit to restrict the powers of the Scottish parliament "will be noted here and across Europe".
"The UK government cannot ignore the reality of devolution or try to drown out what this parliament says," he said.
"They cannot pretend that no motion has been passed.
"It will be the UK that would be breaking trust, and breaking the rules, not us."
Russell has invited British cabinet minister David Lidington to Scotland for last-ditch cross-party talks to break the impasse.
Power grab claim
The SNP, Labour, the Greens and the Liberal Democrats all voted against consent, with only the Scottish Conservatives arguing the changes that have already been made to the bill go far enough to protect devolution.
Scottish officials have accused May of a "power grab".
The British government has accused Scottish opponents of "nit-picking" and has insisted it is trying to protect Britain's economic integrity by building a common framework throughout the country when Brussels regulations are handed over after Brexit.
Sturgeon has rejected claims by May's Scottish Conservative allies that she is trying to "weaponise Brexit" to further her aim of Scottish independence.
Adam Tomkins, the Scottish Conservatives' constitutional spokesman, said: "It's patently obvious that Nicola Sturgeon wants a political crisis to provide cover for her independence drive.
"Sturgeon has refused to compromise. It's not in Scotland's interests that the SNP prefers picking fights to making a deal."
Scotland voted against independence by 55 percent in a referendum in 2014, but Sturgeon insists she has a mandate to hold a second vote since Scotland voted against Brexit by 62 percent in 2016.
"I'm not sure independence will ever be off the table until it is realised," Sturgeon said on Monday.
She has pledged to outline her timetable for a second independence referendum before the end of the year, once the terms of the Brexit agreement become clear.
