Six-times champion Federer had booked his place in the semi-finals with victory over German trailblazer Alexander Zverev but any notion that he would coast through his final group match proved wide of the mark.
Cilic, eliminated after two three-set losses, was hoping to end his year on a high in the city where his Wimbledon dream was ended in this year's final by Federer - the Croatian sobbing on court after suffering blisters.
The world number five survived four break points in the opening set and played an impressive tiebreak.
A stunning angled backhand volley at 5-5 gave him a set point and he crunched away a forehand winner, his 20th of a first set in which Federer's frustration was clear.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
Federer could have been excused for stepping off the gas but the 19-times grand slam champion went up a gear.
The second set was on serve until, at 4-5, Cilic netted a forehand to give Federer a set point which the Swiss converted.
A 100mph forehand service return gave Federer an early break in the decider and Cilic's resistance crumbled quickly as Federer turned on the style to race to victory.
Federer has reached the last four in 14 out of 15 appearances in the year-end tournament and will face David Goffin or Dominic Thiem for a place in the final.
Cilic's career record is now 1-8 at the ATP Finals.
"I was a set and a break points down so happy I found a way out of it," the 36-year-old world number two said.
"I thought try to relax a little bit. This was nice because I could play a bit more freely midway through the second set because I was through regardless.
"But I wanted to keep the momentum going. One last weekend now, one last push, then vacation."
American Jack Sock plays Zverev later on the Thursday and the winner will join Federer in the semis.
(Reporting by Martyn Herman, editing by Ed Osmond)
