The 68-year-old host, famed for his quick wit, sarcasm, offbeat humor, attitude and crazy stunts, hosted Hollywood stars and US presidents in his final weeks of the show before handing the reins to comedian Stephen Colbert.
George Clooney appeared with handcuffs to ensure Letterman would not leave. Comedian Tina Fey stripped off her dress to reveal underwear saying "Bye Dave" and President Barack Obama joked about retirement plans.
Actor Bill Murray, Letterman's first guest on the show, made a final appearance along with singer Bob Dylan on Tuesday night, but the guests for the last show remained a secret.
"It will be a piece of history," Michael Tidwell of Silver Spring, Maryland said as he waited outside the theater on Wednesday morning.
The 59-year-old lawyer was at the first "Late Show with David Letterman" on CBS in 1993, when Letterman moved to the network after losing out to Jay Leno to be host of NBC's "Tonight Show."
Tidwell made the trip to New York because he was among the lucky few with tickets for Letterman's last show.
Photographers and camera crews set up outside the show's home, the Ed Sullivan Theater on Broadway. as tourists and fans took pictures. Around the corner, people milled around West 53rd Street, where Letterman and his guests had performed outdoor stunts, and visited the Hello Deli, regularly featured on the show.
Deli owner Rupert Jee, who became a TV celebrity after appearing more than 200 times in skits and stunts on "Late Show," served coffee and posed for photos with tourists buying souvenirs.
"I don't think anyone can replace Dave. He is so unique and so bold," said Jee, a guest on Tuesday's show.
Tom Barlow, 50, was finishing a series of paintings of the theater with Letterman's name on the marquee to commemorate what he described as the end of an era.
"He was a genius and very down to earth," Barlow said. "My mother, who is 87, is a big fan. She said what she likes about him is that he has an edge. Sometimes it is controversial, but that is what makes him real."
For those feeling a little bit nostalgic, here's a clip on what the Late Night team will miss the most:
