The concluding round-robin match at the year-ender had become a straight knockout between the two players after Rafael Nadal had already sealed top spot in Ilie Nastase Group, and Wawrinka looked the sharper as he won 7-6(4) 6-4.
Earlier, Nadal spent two hours and 36 minutes beating fellow Spaniard David Ferrer 6-7(2) 6-3 6-4 to make sure he went into Saturday's semi-final against world number one Novak Djokovic with a 100 percent record so far this week.
Wawrinka will face Stan Smith Group winner Roger Federer, who like Nadal was unbeaten in group play.
News that makes sense
Your trusted source for staying up-to-date with the world around you. Get free daily news updates and analysis, straight to your inbox.
Murray and Wawrinka were both guilty of numerous careless errors, Murray in a first-set tiebreak that he led 4-2.
His Swiss opponent then wobbled towards the end and nearly let slip a 5-2 lead, having to save two break points when he served for the match for a second time at 5-4.
World number two Murray, who smashed a frame in anger in the final game, said the proximity of next week's Davis Cup final on a claycourt in Ghent had not been a factor in his defeat.
"Way too many errors. Cheap ones, as well. Disappointing," the 28-year-old, who fell at the group stage for the second year running, told reporters.
"I didn't find it difficult not thinking about the Davis Cup final, to be honest. Obviously the only positive for me this week is I've come away from it injury-free.
"Now I have a couple more days to get ready for Belgium."
Murray was straight back on court afterwards, taking out his frustration in a knockup with coach Jonas Bjorkman on a deserted centre court as cleaners picked up litter in the stands.
With Nadal already in the semi-finals and Ferrer going home after two defeats, the clash of the Spaniards had no impact on the tournament but the packed arena was treated to the longest match of the tournament so far.
It looked as though Nadal might be in for an easy ride when he won 10 of the first 11 points, breaking Ferrer twice, but his 33-year-old opponent is renowned as one of the sport's best battlers and he hit back to win a tiebreak.
World number five Nadal, who had already beaten fourth-ranked Wawrinka and Murray, buckled down though and levelled the match before breaking the Ferrer serve in the final game of the contest.
"It should be like this," Nadal, who has been transformed since a modest summer, told reporters.
"Doesn't matter if I was qualified. For me the main goal is to try to arrive in good shape next year. Today was an opportunity to play against another of the top eight players of the world.
"I think I competed well. I won a very tough match. I think a very good match in terms of quality moments."
In the doubles, Ivan Dodig and world number one Marcelo Melo beat Marcin Matkowski and Nenad Zimonjic to set up a semi-final against Rohan Bopanna and Florin Mergea.
Second seeds Jean-Julien Rojer and Horia Tecau will face the Bryan twins after beating French duo Pierre-Hugues Herbert and Nicolas Mahut.
(Reporting by Martyn Herman; Editing by Ken Ferris/Alan Baldwin)
