The son of Ireland's only Tour de France winner, Stephen Roche, ended up third behind eighth stage winner Leopold Koenig of the Czech Republic and Spain's Dani Moreno on the 14-km Alto de Penas Blancas summit finish.
The Saxo-Tinkoff rider now leads overall by 17 seconds from American Chris Horner. Moreno is third while Italian Nibali, 16th in the 158.6-km leg from Jerez de la Frontera, occupies fourth position.
"I benefited from a bit of an opening when things stalled in the main pack behind Koenig," said Roche, Ireland's first leader since Sean Kelly won the race in 1988.
"I was only eight seconds behind Nibali before the stage and I've never led a Grand Tour before so I wanted to go all out and give it absolutely everything," he told reporters.
"Right now I'm the happiest man on earth, it's been an incredible week for me," added Roche who won his first Grand Tour stage on the Vuelta's first summit finish last Sunday and also leads the King of the Mountains competition.
With two summit finishes looming on Sunday and Monday, the 29-year-old was cautious about his prospects of remaining in top spot.
"It would be pretty optimistic to say I can hold this lead all the way to Madrid," said Roche whose best Grand Tour finish is seventh in the 2010 Vuelta.
"Monday is one of the hardest stages of the entire race and if I can get through these three days in the lead that would be great."
Stage winner Koenig's small-scale NetApp Endura team are one of four non-World Tour squads to benefit from an invite to the Vuelta.
"This climb suited me down to the ground, its steeper segments were all in the early part. Once I got through that with the favourites I thought my chances were pretty good," said the Czech.
"I saw Anton was suffering and managed to catch him just before the line. I'll try for the overall lead now but anything I get from here on is a bonus for me and the team."
The Vuelta's second straight mountain-top finish on Sunday is a long climb with lung-burstingly steep segments of up to 25 percent in the sierras of northern Andalusia.
The race ends in Madrid on September 15.
(Editing by Tony Jimenez)
