McGill was 10-6 ahead as the pair went into the final session and he secured the three frames he needed to reach the quarter-finals in his first appearance at the Crucible Theatre.
The Scot took the first frame after the resumption with a break of 68 and although Selby seized the next, an 87 put his opponent 12-7 ahead.
Englishman Selby then threatened to mount a fightback but McGill held firm and will next meet 2005 champion Shaun Murphy or Joe Perry.
Selby's defeat continued the so-called 'Crucible Curse'. No first-time champion has ever made a successful defence of his title.
"I have no embarrassment in losing," said Selby. "He played fantastic all match."
(Reporting by Douglas Beattie, editing by Tony Jimenez)
Share
