Hogg, player of the tournament in the Six Nations for the last two years, was one of the favourites to play fullback against the All Blacks in the Lions' three-test series starting on June 24.
One of only two Scots named in coach Warren Gatland's original 41-man squad, Hogg was caught in the face by team mate Connor Murray's elbow as he chased his own kick in the 12-3 victory in Christchurch last Saturday.
Scans confirmed a facial bone fracture and Hogg will return home for further treatment, the first of the squad forced out of the tour by injury. He will not be immediately replaced.
"It is hugely disappointing for Stuart to have to return home early and for us to lose one of our touring party," Gatland said in a news release.
"Stuart has been a key member of our squad on and off the field and it is disappointing to see injury cut short his time with us in New Zealand."
Welshman Leigh Halfpenny, player of the test series on the successful 2013 tour of Australia, now looks certain to take the number 15 jersey for the matches against the All Blacks.
Halfpenny does not offer as much attacking threat as the aggressive running Hogg but is one of the best place-kickers in European rugby.
Jared Payne has a chance to put his hand up for the test side when the versatile New Zealand-born Ireland international plays fullback against the Otago Highlanders in Dunedin later on Tuesday.
Hogg, who also toured Australia four years ago but did not play in any of the tests, featured in this year's tour opener against the New Zealand Barbarians as well as against the Crusaders.
"I'm gutted to be leaving the tour at this stage," he said.
"It is an honour to have been involved in a second British & Irish Lions tour and I have hugely enjoyed my time with the squad and I'm very excited about the potential of this group."
Before the Lions departed for New Zealand they lost number eight Billy Vunipola to injury, while his England team mate Ben Youngs pulled out of the tour for family reasons and was replaced by Scottish scrumhalf Greig Laidlaw.
(Reporting by Nick Mulvenney; Editing by Peter Rutherford)
