The early exit also denied world number two Nadal a chance to take over from the absent Andy Murray at the top of the rankings, a position the Spaniard would have regained for the first time since 2014 had he advanced to the semi-finals.
Shapovalov showed no sign of nerves in their first ever meeting, the left-hander playing the more aggressive tennis by utilising his huge forehand on roundhouse swings and frequently attacking the net.
The teenager's quixotic bid looked doomed in the third set tiebreaker when he fell 3-0 behind but the world number 143 refused to buckle and only coughed up one more point in the decider.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
As a reward for his unlikely victory, the 2016 Wimbledon junior champion will next face 29-year-old Frenchman Adrian Mannarino in the quarter-finals on Friday.
Elsewhere, world number three Roger Federer dropped his first set since June but the Swiss recovered to dispatch Spaniard David Ferrer 4-6 6-4 6-2 in their last 16 meeting.
Federer was sloppy from the outset on centre court, sending routine forehands wide and providing the underdog Ferrer with early break opportunities in the Montreal encounter.
Ferrer capitalised by winning the opener to end Federer's streak of 32 consecutive sets won, a record that dates back to his defeat to Tommy Haas at the Stuttgart Open on June 14.
"He had a good start, I had a better finish," Federer told reporters after the match.
Frustration boiled over for the Australian Open and Wimbledon champion in the second set after he misfired with an overhead smash.
Federer, who turned 36 this week, responded by hitting a ball into the stands, earning a reprimand but no official ball abuse call from the umpire.
The outburst appeared to steady the Swiss, whose serve and overall accuracy improved as the two-hour match progressed.
Ferrer has now lost all of his 17 contests against Federer, who is also vying to take the world number one spot from the injured Murray.
Next up for Federer is a quarter-final against 12th-seeded Spaniard Roberto Bautista Agut, who defeated Frenchman Gael Monfils 4-6 7-6(5) 7-6(2) in a marathon tussle.
In other tournament action, American Jared Donaldson lost to Diego Schwartzman 0-6 7-5 7-5, while his countryman Sam Querrey was thrashed 6-4 6-1 by Kevin Anderson of South Africa.
Australian Nick Kyrgios fell 6-4 6-3 to fourth seed Alexander Zverev as the German finally claimed victory over his opponent in their third meeting of the season.
(Reporting by Rory Carroll in Los Angeles; Editing by John O'Brien)
