Brazilian Italo Ferreira beat the three-time world champion 15.66 to 12.83 in an engrossing final to deny Fanning a fairytale finish.
It is the first title win on the world tour for Ferreira, while Fanning ends his career alongside Kelly Slater and Mark Richards as a four-time Bells champion.
Related reading
'He's an icon': Mick Fanning paddles out one last time
Fanning would have finished his career as the world No.1 had he beaten Ferreira, but the Australian is happy heading into retirement.
The finalists shared a warm embrace on their boards as the second ticked down in the final before they were mobbed by the crowd as they returned to shore.
Fanning's retirement made him the No.1 story of this year's event.
"It's really special, it's a very special moment," Fanning said.
"Just to see how much it meant to him (Ferreira), it made me feel better.
"It has been an amazing career, I want to thank everyone along the way.
"I walk away with so many great memories."
Related reading
Stephanie Gilmore wins Bells Beach surfing title
Ferreira recovered from a couple of falls early to win 15.66 to 12.83, with his final wave score putting him in front.
Ferreira is the second Brazilian to win the Bells Beach title, after Adriano De Souza in 2013.
Earlier on Thursday, Australian six-time world champion Stephanie Gilmore won her fourth Rip Curl Pro title when she beat Tatiana Weston-Webb in a nail-biting final.
Weston-Webb thought she had snatched the lead from Gilmore with her wave in the last minute of the final.
She needed a 6.80 to win but the judges awarded the Hawaiian a 6.57.
